VICTORIA

UNIVERSITY

OF

WELLINGTON

Te

Whare

Wananga

o

te

Upoko

o

te

Ika

a

Maui

VUW

Department

of

Computer

Science

PO

Box

Wellington

New

Zealand

Tel

Fax

Internet

Tech.Reports@comp.vuw.ac.nz

ART-based

Modular

Networks

for

Incremental

Learning

of

Hierarchical

Clusterings

Guszti

Bartfai

and

Roger

White

Technical

Report

August

Abstract

This

paper

introduces

HART-S

a

new

modular

neural

network

that

can

incrementally

learn

stable

hierarchical

clusterings

of

arbitrary

sequences

of

input

patterns

by

self-organisation

The

network

is

a

cascade

of

Adaptive

Resonance

Theory

-LRB-

ART

-RRB-

modules

in

which

each

module

learns

to

cluster

the

differences

between

the

input

pattern

and

the

selected

category

prototype

at

the

previous

module

Input

patterns

are

first

classified

into

a

few

broad

categories

and

successive

ART

modules

find

increasingly

specific

categories

until

a

threshold

is

reached

the

level

of

which

can

be

controlled

by

a

global

parameter

called

resolution

The

network

thus

essentially

implements

a

divisive

-LRB-

or

splitting

-RRB-

hierarchical

clustering

algorithm

hence

the

name

HART-S

-LRB-

for

Hierarchical

ART

with

Splitting

-RRB-

HART-S

is

also

compared

and

contrasted

to

HART-J

-LRB-

for

Hierarchical

ART

with

Joining

-RRB-

another

variant

that

was

proposed

earlier

by

the

first

author

The

network

dynamics

are

specified

and

some

useful

properties

of

both

networks

are

given

and

then

proven

Experiments

were

carried

out

on

benchmark

datasets

to

demonstrate

the

representational

and

learning

capabilities

of

both

networks

and

to

compare

the

developed

clusterings

with

those

of

two

classical

methods

and

a

conceptual

clustering

algorithm

A

brief

survey

of

related

neural

network

models

is

also

provided

Publishing

Information

This

paper

will

appear

in

a

Special

Issue

of

Connection

Science

on

Combining

Artificial

Neural

Networks

Author

Information

Guszti

Bartfai

is

a

lecturer

at

the

Department

of

Computer

Science

Roger

White

is

working

on

his

MSc

thesis

under

the

supervision

of

Guszti

Bartfai

The

authors

can

be

contacted

through

the

department

or

directly

by

e-mail

at

the

guszti@comp.vuw.ac.nz

address

Introduction

The

combination

of

neural

networks

has

gained

increasing

attention

in

recent

years

Although

some

monolithic

black-box

like

networks

like

the

multi-layer

perceptron

have

been

shown

to

be

universal

function

approximators

and

used

in

many

practical

applications

they

have

some

well-known

disadvantages

the

most

severe

of

those

is

intractable

training

time

One

way

to

alleviate

this

problem

is

to

divide

the

task

into

smaller

ones

and

train

the

individual

and

proportionally

smaller

networks

on

each

sub-task

separately

This

modular

approach

appears

to

be

supported

by

findings

in

brain

study

which

suggest

that

the

human

brain

is

not

just

one

big

mass

of

neurons

For

example

neurons

of

the

neocortex

where

most

high-level

cognitive

processes

occur

are

of

few

types

and

their

connections

are

organised

in

relatively

few

schemes

or

basic

circuits

-LRB-

Shepherd

-RRB-

The

primate

visual

system

has

also

been

found

to

be

highly

modularised

with

extensively

interconnected

subdivisions

performing

linear

and

non-linear

filtering

object

recognition

and

so

on

-LRB-

Van

Essen

et

al.

-RRB-

The

arguments

for

decomposition

have

engineering

complexity

and

psychological

aspects

as

well

-LRB-

Hrycej

-RRB-

There

are

also

a

number

of

theoretical

advantages

coding

can

be

sparse

scalability

of

solutions

modularity

of

solution

leading

to

self-similar

fractallike

architectures

-LRB-

Happel

and

Murre

-RRB-

Consequently

modular

neural

network

models

have

been

proposed

-LRB-

see

for

example

-LRB-

Hecht-Nielsen

Moody

and

Darken

-RRB-

-RRB-

which

possess

distinctive

advantages

over

monolithic

ones

like

faster

training

-LRB-

LeCun

et

al.

-RRB-

and

better

overall

performance

-LRB-

Jordan

and

Jacobs

-RRB-

Besides

smaller

computational

costs

modular

networks

can

also

be

used

to

solve

more

complex

tasks

that

simply

can

not

be

done

-LRB-

i.e.

represented

and

learned

-RRB-

by

a

single

network

module

One

such

area

is

unsupervised

clustering

which

is

the

main

focus

of

our

paper

The

ability

to

learn

about

the

environment

without

a

teacher

has

long

been

considered

an

important

characteristic

of

intelligent

systems

Unsupervised

learning

can

be

found

both

at

sensory-level

of

mammals

and

at

higher

cognitive

levels

of

humans

Therefore

it

has

been

an

important

topic

in

neural

network

research

-LRB-

e.g.

-LRB-

Kohonen

-RRB-

-LRB-

Oja

-RRB-

-LRB-

Rumelhart

et

al.

-RRB-

-RRB-

in

recent

years

Unsupervised

learning

networks

typically

perform

dimensionality

reduction

or

pattern

clustering

In

the

latter

case

upon

presentation

of

an

input

pattern

the

node

whose

connection

weight

vector

is

the

closest

to

the

current

input

vector

-LRB-

in

some

distance

metrics

-RRB-

will

become

the

only

active

node

-LRB-

winner-take-all

competition

-RRB-

and

will

be

allowed

to

learn

i.e.

modify

its

connection

weights

After

repeated

exposure

to

the

input

environment

the

network

will

store

a

prototypical

element

of

each

category

or

class

in

its

connections

weights

The

number

and

size

of

classes

the

network

finds

on

a

given

training

set

depends

on

the

number

of

output

nodes

The

more

neurons

the

network

has

the

larger

number

of

more

specific

categories

it

will

find

Because

of

its

winner-take-all

characteristics

the

network

will

only

be

able

to

find

one

category

for

a

given

input

pattern

regardless

of

how

large

the

network

is

However

environments

are

often

more

complex

and

may

exhibit

a

hierarchical

structure

-LRB-

e.g.

taxonomy

of

animals

-RRB-

which

humans

can

learn

without

apparent

difficulty

It

is

therefore

reasonable

to

assume

that

network

circuitry

which

performs

hierarchical

clustering

is

present

in

the

brain

An

example

for

such

a

network

is

discussed

in

-LRB-

Ambros-Ingerson

et

al.

-RRB-

where

it

was

suggested

that

the

paleocortex

performs

hierarchical

clustering

The

authors

also

hypothesise

that

networks

and

circuitry

of

similar

design

in

other

brain

regions

contain

computational

elements

sufficient

to

construct

perceptual

hierarchies

for

use

in

recognizing

environmental

cues

and

suggest

the

possibility

that

this

biologically

generated

mechanism

for

hierarchical

clustering

may

be

a

routine

part

of

perceptual

recognition

memory

behavior

in

animals

and

humans

It

is

therefore

worthwhile

investigating

neural

network

architectures

that

can

learn

class

hierarchies

Networks

that

contain

multiple

competitive

layers

appear

to

be

suitable

candi

dates

For

example

Rumelhart

et

al.

-LRB-

Rumelhart

et

al.

-RRB-

suggest

that

a

hierarchical

structure

can

be

formed

if

a

network

has

several

inhibitory

clusters

in

each

layer

that

work

in

parallel

and

each

of

them

receives

input

from

the

output

of

the

previous

layer

only

-LRB-

which

is

a

vector

with

the

winners

in

each

active

cluster

-RRB-

This

method

was

shown

to

have

some

advantages

on

an

artificial

example

but

it

is

not

clear

due

to

its

inherently

probabilistic

nature

whether

it

can

be

used

successfully

in

other

possibly

more

complex

tasks

Standard

competitive

learning

networks

also

have

several

drawbacks

like

long

training

time

finding

and

getting

stuck

in

suboptimal

solutions

and

instability

against

changes

in

the

environment

An

alternative

model

that

is

based

on

solid

neurophysiological

and

cognitive

principles

and

also

overcomes

the

above

problems

is

the

Adaptive

Resonance

Theory

-LRB-

ART

-RRB-

neural

network

-LRB-

Carpenter

and

Grossberg

-RRB-

They

are

more

suitable

for

the

systematic

construction

of

modular

networks

because

of

their

well-defined

interfaces

as

well

as

features

that

most

other

networks

lack

In

specific

ART

networks

have

the

ability

to

create

new

output

nodes

-LRB-

i.e.

categories

-RRB-

dynamically

and

do

not

suffer

from

the

problem

of

forgetting

previously

learned

categories

if

the

environment

changes

In

other

words

it

provides

a

solution

to

the

stability-plasticity

dilemma

-LRB-

Carpenter

and

Grossberg

-RRB-

A

neural

network

with

similar

functionalities

-LRB-

CALM

for

CAtegorizing

and

Learning

Module

-RRB-

was

proposed

as

a

computational

primitive

in

more

complex

modular

networks

-LRB-

Happel

and

Murre

-RRB-

Despite

their

useful

properties

however

ART

networks

by

themselves

can

only

learn

input

categories

at

a

given

level

of

specificity

-LRB-

i.e.

partition

the

input

space

-RRB-

which

depends

on

a

global

parameter

called

vigilance

The

central

idea

of

this

paper

is

to

combine

ART

networks

in

a

systematic

way

into

a

modular

architecture

which

is

capable

of

learning

class

hierarchies

from

arbitrary

sequences

of

input

patters

while

retaining

the

main

properties

of

single

ART

networks

namely

autonomous

fast

and

stable

learning

by

self-organisation

In

specific

we

introduce

a

new

ART-based

modular

network

that

develops

cluster

hierarchies

by

first

creating

large

general

classes

which

are

subsequently

refined

in

lower

layers

by

splitting

up

each

super-class

into

sub-classes

-LRB-

by

clustering

features

that

are

different

from

the

super-class

-RRB-

The

biological

plausibility

of

such

mechanism

was

suggested

in

-LRB-

Ambros-Ingerson

et

al.

-RRB-

We

call

this

splitting

a

term

used

in

statistical

clustering

after

-LRB-

Hartigan

-RRB-

hierarchical

ART

network

or

hart-s

-LRB-

for

Hierarchical

ART

with

Splitting

-RRB-

In

particular

we

contrast

its

functionality

and

properties

to

those

of

the

joining

hierarchical

ART

network

-LRB-

hart-j

-RRB-

which

was

proposed

independently

by

Bartfai

-LRB-

Bartfai

-RRB-

and

Ishihara

-LRB-

Ishihara

et

al.

-RRB-

under

the

names

HART

and

arboART

respectively

-LRB-

We

discuss

some

related

works

in

more

detail

in

section

-RRB-

The

hart-j

network

has

several

layers

of

ART

modules

where

each

layer

learns

to

cluster

the

prototypes

the

previous

layer

has

developed

The

network

therefore

builds

successively

larger

classes

in

upper

layers

by

joining

the

lower

level

more

specific

ones

Section

describes

the

ART

network

and

summarises

its

properties

The

hierarchical

ART

models

are

introduced

in

section

The

joining

hierarchical

ART

model

-LRB-

hart-j

-RRB-

and

its

properties

are

discussed

in

sections

and

respectively

The

splitting

hierarchical

ART

model

-LRB-

hart-s

-RRB-

and

its

properties

are

discussed

in

sections

and

respectively

The

experiments

that

were

carried

out

on

a

machine

learning

database

are

shown

in

section

and

the

results

are

discussed

in

section

Finally

conclusions

are

drawn

in

section

The

ART

network

This

section

describes

the

ART

architecture

at

a

level

of

detail

that

is

necessary

for

understanding

the

rest

of

the

paper

Unless

otherwise

stated

we

refer

to

the

binary

version

of

ART

-LRB-

Carpenter

and

Grossberg

-RRB-

Adaptive

Resonance

Theory

-LRB-

ART

-RRB-

architectures

are

neural

networks

that

develop

stable

recognition

codes

by

self-organisation

in

response

to

arbitrary

sequences

of

input

patterns

They

were

designed

to

solve

the

stability-plasticity

dilemma

that

every

intelligent

machine

learning

system

has

to

face

how

to

keep

learning

from

new

events

without

forgetting

previously

learned

information

An

ART

network

is

built

up

of

three

layers

the

input

layer

-LRB-

-RRB-

the

comparison

layer

-LRB-

-RRB-

and

the

recognition

layer

-LRB-

-RRB-

with

M

M

and

N

neurons

respectively

-LRB-

see

figure

-RRB-

The

input

layer

stores

the

input

pattern

and

each

neuron

in

the

input

layer

is

connected

to

its

corresponding

node

in

the

comparison

layer

via

one-to-one

non-modifiable

links

Nodes

in

the

layer

represent

input

categories

The

and

layers

interact

with

each

other

through

weighted

bottom-up

and

top-down

connections

that

are

modified

when

the

network

learns

The

and

R

signals

control

the

operation

of

the

network

the

non-specific

gain

control

signals

and

are

used

to

enable/disable

all

the

nodes

in

the

and

layers

respectively

The

main

role

of

is

to

implement

template

matching

in

the

layer

-LRB-

see

equation

-LRB-

-RRB-

below

where

use

of

the

reset

signal

R

is

also

described

-RRB-

The

learning

process

of

the

network

can

be

described

as

follows

At

each

presentation

of

a

non-zero

binary

input

pattern

x

-LRB-

xi

i

M

-RRB-

the

network

attempts

to

classify

it

into

one

of

its

existing

categories

based

on

its

similarity

to

the

stored

prototype

of

each

category

node

More

precisely

for

each

node

i

in

the

layer

the

bottom-up

activation

Tj

is

calculated

which

can

be

expressed

as

Tj

jwj

xj

jwjj

j

N

-LRB-

-RRB-

where

j

j

is

the

norm

operator

-LRB-

jxj

PMi

xi

-RRB-

wj

is

the

-LRB-

binary

-RRB-

weight

vector

-LRB-

or

prototype

-RRB-

of

category

j

and

is

the

choice

parameter

-LRB-

Carpenter

et

al.

-RRB-

Then

the

node

J

that

has

the

highest

bottom-up

activation

i.e.

TJ

maxfTj

jj

Ng

is

selected

-LRB-

winner-take-all

competition

-RRB-

The

weight

vector

of

the

winning

node

-LRB-

wJ

-RRB-

will

then

be

compared

to

the

current

input

at

the

comparison

layer

If

they

are

similar

enough

i.e.

they

satisfy

the

jwJ

xj

jxj

ae

-LRB-

-RRB-

matching

condition

where

ae

is

a

system

parameter

called

vigilance

-LRB-

ae

-LSB-

-RSB-

-RRB-

node

J

will

capture

the

current

input

and

the

network

learns

by

modifying

wJ

wnew

J

-LRB-

wold

J

x

-RRB-

-LRB-

-RRB-

wold

J

-LRB-

-RRB-

where

is

the

learning

rate

-LRB-

-LSB-

-RSB-

-RRB-

All

other

weights

in

the

network

remain

unchanged

If

however

the

stored

prototype

wJ

does

not

match

the

input

sufficiently

i.e.

condition

-LRB-

-RRB-

is

not

met

the

winning

node

will

be

reset

-LRB-

by

activating

the

reset

signal

R

in

figure

-RRB-

for

the

period

of

presentation

of

the

current

input

Then

another

node

-LRB-

or

category

-RRB-

is

selected

with

the

highest

Tj

whose

prototype

will

be

matched

against

the

input

and

so

on

This

hypothesis-testing

cycle

is

repeated

until

the

network

either

finds

a

stored

are

ART

networks

that

accept

both

continuous

and

binary

inputs

-LRB-

Carpenter

and

Grossberg

Carpenter

et

al.

-RRB-

the

bottom-up

weights

are

essentially

normalised

versions

of

the

corresponding

-LRB-

binary

-RRB-

top-down

ones

subsuming

them

into

a

single

weight

vector

and

modifying

the

network

equations

accordingly

is

a

commonly

used

simplification

which

we

shall

adopt

throughout

the

paper

This

also

assumes

the

network

is

in

fast

learning

mode

-LRB-

see

below

-RRB-

subsystem

R

Orienting

Attentional

subsystem

Input

m-m

W

Figure

Architecture

of

the

ART

network

Thick

arrows

denote

modifiable

connections

The

and

signs

represent

excitatory

and

inhibitory

connections

respectively

The

symbols

mg

mg

represent

fone,manyg-to-fone

manyg

connections

Winning

category

nodes

are

selected

by

the

attentional

subsystem

Category

search

is

controlled

by

the

orienting

subsystem

If

the

degree

of

category

match

at

the

layer

is

lower

than

the

vigilance

level

ae

the

non-specific

reset

signal

R

will

be

triggered

which

will

deactivate

the

winning

node

for

the

period

of

presentation

of

the

current

input

category

whose

prototype

matches

the

input

well

enough

or

allocates

a

new

node

Then

learning

takes

place

according

to

-LRB-

-RRB-

It

is

important

to

note

that

once

a

category

is

found

the

comparison

layer

-LRB-

-RRB-

holds

wJ

x

until

the

current

input

is

removed

It

can

be

shown

that

after

an

initial

period

of

self-stabilisation

the

network

will

directly

-LRB-

i.e.

without

search

-RRB-

access

the

prototype

of

one

of

the

categories

it

has

found

in

a

given

training

set

Note

also

however

that

as

a

consequence

of

its

stability-plasticity

property

the

network

is

capable

of

learning

on-line

i.e.

refining

its

learned

categories

in

response

to

a

stream

of

input

patterns

-LRB-

as

opposed

to

being

trained

off-line

on

a

finite

training

set

-RRB-

The

set

of

attribute

values

described

by

ART

cluster

prototypes

correspond

to

the

intersection

of

all

those

attribute

values

found

in

instances

of

that

cluster

In

terms

of

the

clustering

algorithm

this

is

not

as

restrictive

as

it

first

seems

A

common

technique

used

with

ART

is

the

complement

coding

of

input

patterns

-LRB-

Carpenter

et

al.

-RRB-

The

result

of

using

this

together

with

encoded

attribute

values

is

that

one

half

of

a

cluster

prototype

corresponds

to

the

intersection

of

the

attribute

values

that

all

instances

in

that

cluster

do

not

have

or

from

another

angle

the

union

of

the

attribute

values

present

in

that

cluster

In

total

then

a

cluster

prototype

specifies

a

range

of

values

for

each

attribute

and

has

an

effect

which

combined

with

Fuzzy

ART

-LRB-

Carpenter

et

al.

-RRB-

adds

significantly

to

the

clustering

capability

of

the

mechanism

The

number

of

developed

categories

can

be

controlled

by

setting

ae

the

higher

the

vigilance

level

the

larger

number

of

more

specific

categories

will

be

created

-LRB-

If

ae

the

network

will

create

a

new

category

for

every

unique

input

-RRB-

In

order

to

get

two

or

more

views

of

the

training

set

with

the

same

network

one

can

consider

choosing

low

vigilance

first

to

get

an

overall

view

and

then

raise

it

at

a

later

phase

during

training

to

gain

a

more

detailed

view

as

well

This

however

will

result

in

the

network

not

being

able

to

access

the

overall

view

-LRB-

i.e.

coarser

categorisation

-RRB-

anymore

but

retaining

only

the

more

specific

categories

even

if

vigilance

is

lowered

again

later

This

is

because

only

one

category

is

selected

for

a

given

input

and

if

two

categories

-LRB-

wl

wh

-RRB-

were

found

to

match

the

current

input

perfectly

i.e.

jwl

xj

jwlj

and

jwh

xj

jwhj

the

more

specific

one

-LRB-

i.e.

of

the

larger

norm

-RRB-

will

be

preferred

due

to

-LRB-

-RRB-

There

is

no

relationship

between

any

pairs

of

category

prototypes

except

that

they

are

in

the

same

network

and

compete

with

each

other

The

network

therefore

with

its

single

layer

of

category

nodes

is

not

capable

of

representing

-LRB-

and

thus

learning

-RRB-

a

hierarchy

of

classes

This

motivated

us

to

design

networks

with

multiple

ART

modules

Hierarchical

ART

networks

This

section

discusses

two

ART-based

architectures

that

can

perform

hierarchical

clustering

by

incremental

learning

Both

of

them

are

essentially

a

cascade

of

ART

modules

with

the

input

pattern

entering

at

one

end

Information

processed

by

an

ART

module

is

passed

on

to

the

next

one

-LRB-

see

figure

-RRB-

At

this

level

it

is

basically

the

kind

of

information

that

is

passed

on

that

distinguishes

the

two

models

The

joining

HART

network

-LRB-

hart-j

-RRB-

first

clusters

input

patterns

into

-LRB-

potentially

many

-RRB-

specific

categories

-LRB-

with

high

vigilance

level

in

the

first

module

-RRB-

and

then

successive

layers

cluster

the

prototypes

that

are

developed

at

the

previous

layer

This

architecture

was

first

presented

in

-LRB-

Bartfai

-RRB-

Here

we

describe

its

main

features

as

well

as

some

further

properties

some

of

which

are

particularly

useful

when

comparing

them

to

those

of

the

Module

LModule

Input

Pattern

Figure

Overall

view

of

the

architecture

of

a

Hierarchical

ART

network

Each

module

is

an

ART

network

splitting

HART

network

The

hart-s

network

first

creates

a

few

typically

broad

clusters

Then

successive

layers

learn

to

cluster

the

differences

between

a

prototype

and

the

input

vector

at

the

previous

layer

The

joining

hierarchical

ART

network

-LRB-

hart-j

-RRB-

This

model

is

based

on

the

idea

of

starting

with

the

most

specific

clustering

of

the

input

domain

and

progressively

merging

such

clusters

into

more

general

ones

Only

the

similarity

between

an

input

to

an

ART

module

and

the

prototype

of

the

category

of

that

input

is

propagated

to

the

next

ART

module

The

input

signal

to

the

first

module

is

the

similarity

between

the

input

pattern

and

its

most

specific

categorisation

-LRB-

itself

-RRB-

so

is

equivalent

to

the

input

pattern

itself

This

similarity

is

exactly

the

content

of

the

layer

during

resonance

so

the

structure

of

hart-j

reflects

this

with

two

connections

between

successive

ART

modules

-LRB-

figure

-RRB-

One

connection

propagates

inhibitory

signals

from

global

reset

to

the

gain

controls

of

a

second

module

so

that

it

can

not

become

active

until

a

resonant

state

is

obtained

in

the

first

The

other

connection

propagates

the

contents

of

the

layer

of

the

first

module

to

the

layer

of

the

second

More

formally

the

hart-j

network

is

described

by

the

following

parameter

tuple

HJ

fL

M

N

ae

g

where

the

meaning

of

each

parameter

is

as

follows

L

the

number

of

ART

modules

used

M

-LRB-

ML

-RRB-

the

size

of

each

and

layer

N

-LRB-

NL

-RRB-

the

size

of

each

layer

-LRB-

L

-RRB-

the

choice

parameter

of

each

ART

module

-LRB-

k

typically

k

o

-RRB-

ae

-LRB-

aeL

-RRB-

the

vigilance

level

of

each

ART

module

aek

-LSB-

-RSB-

-LRB-

L

-RRB-

the

learning

rate

of

each

ART

module

k

-LSB-

-RSB-

with

the

architectural

constraint

of

Mk

M

for

k

L.

For

further

simplicity

however

we

will

also

assume

Nk

N

k

and

k

-LRB-

i.e.

fast

learning

-RRB-

for

k

L

So

it

is

only

the

aek

vigilance

levels

that

are

allowed

to

be

different

in

each

ART

module

From

figure

and

noting

the

contents

of

the

layer

in

resonance

-LRB-

see

section

-RRB-

the

local

input

to

ART

module

k

can

be

expressed

as

real-time

implementations

this

will

involve

specifying

the

dynamics

of

these

connections

so

as

to

filter

out

undesirable

spikes

of

the

reset

signal

during

category

search

R

R

G

G

R

G

Input

Figure

Architecture

of

a

three-layer

hart-j

network

Connections

within

the

ART

modules

are

not

shown

The

and

ART

signals

are

represented

with

one

G

signal

for

each

module

-LRB-

Gi

i

-RRB-

The

meanings

of

the

and

symbols

are

explained

at

figure

xk

xk

wk

J

k

L

-LRB-

-RRB-

where

k

J

is

the

index

of

the

winning

node

in

ART

module

k

The

prototype

-LRB-

or

weight

vector

-RRB-

of

the

winning

category

k

J

in

layer

k

after

training

-LRB-

see

equation

-LRB-

-RRB-

if

-RRB-

becomes

wk

J

xk

wk

J

k

L

-LRB-

-RRB-

Applying

-LRB-

-RRB-

and

-LRB-

-RRB-

to

layers

k

k

recursively

-LRB-

and

ignoring

the

indices

of

winning

nodes

for

simplicity

-RRB-

we

get

wk

wk

k

L

-LRB-

-RRB-

where

is

the

local

input

to

the

first

ART

module

which

is

also

the

global

input

X

to

the

entire

hart-j

network

Equation

-LRB-

-RRB-

can

then

be

written

as

wk

X

wk

X

Wk

k

L

-LRB-

-RRB-

where

Wk

-LRB-

or

more

precisely

Wk

J

-RRB-

is

the

global

-LRB-

or

hierarchical

-RRB-

prototype

of

level

k

i.e.

the

prototype

of

category

J

in

level

k

with

respect

to

the

global

input

X.

From

-LRB-

-RRB-

and

-LRB-

-RRB-

it

can

be

seen

that

wk

wk

wk

-LRB-

-RRB-

which

shows

that

the

network

develops

more

general

categories

at

successively

higher

levels

It

also

follows

from

-LRB-

-RRB-

and

-LRB-

-RRB-

that

Wk

wk

k

L

-LRB-

-RRB-

Since

the

hart-j

network

develops

more

general

hierarchical

categories

it

is

instructive

to

ask

what

the

maximum

degree

of

specificity

any

level

k

category

can

have

given

a

sequence

of

training

patterns

of

size

S

and

-LRB-

local

-RRB-

vigilance

levels

aek

In

a

single

ART

network

this

is

equivalent

to

asking

what

the

minimum

level

of

generality

of

any

category

will

be

which

is

simply

given

by

the

-LRB-

only

-RRB-

vigilance

level

ae

since

jwJ

j

aejxj

-LRB-

aeS

-RRB-

for

any

input

x

and

winning

node

J

For

a

given

level

in

a

hart-j

network

however

the

two

limits

-LRB-

i.e.

minimum

level

of

generality

and

maximum

level

of

specificity

-RRB-

are

separated

since

the

local

inputs

to

each

level

will

vary

due

to

the

generalisation

done

by

previous

modules

So

in

a

hart-j

network

there

will

be

a

region

of

effective

vigilance

at

level

k

which

provides

bounds

of

the

specificity

that

level

can

have

with

respect

to

the

global

input

X

A

discussion

on

the

lower

bound

of

this

region

can

be

found

in

-LRB-

Bartfai

-RRB-

Here

we

provide

the

upper

bound

-LRB-

k

max

-RRB-

which

is

of

more

use

for

practical

considerations

-LRB-

For

example

it

allows

us

to

put

a

bound

on

the

number

of

ART

modules

needed

in

an

application

by

guaranteeing

a

given

level

of

specificity

that

level

k

can

not

exceed

-RRB-

More

specifically

in

each

level

we

look

for

the

smallest

value

z

such

that

jwkj

jxkj

aek

jWkj

jXj

z

-LRB-

-RRB-

The

value

of

k

max

is

given

with

proof

in

section

Dynamics

of

hart-j

The

dynamics

of

the

hart-j

network

are

closely

linked

to

those

of

component

ART

modules

and

are

described

as

follows

For

i

M

j

N

and

k

L

ffl

Initialisation

The

initial

values

of

all

the

connection

weights

are

wk

ji

ffl

Layer

activity

vectors

The

layer

xk

The

layer

-LRB-

xk

wk

j

when

node

j

is

active

and

module

k

is

in

resonance

xk

otherwise

The

layer

j

-LRB-

when

node

j

is

active

and

module

k

is

in

resonance

otherwise

Whenever

an

input

is

presented

and

module

k

is

in

resonance

all

nodes

are

enabled

and

an

initial

category

choice

is

made

here

are

assumed

to

be

of

constant

norm

only

for

the

sake

of

simplicity

The

discussion

will

still

hold

with

minor

modifications

if

the

inputs

are

not

normalised

ffl

Category

Choice

For

any

module

k

if

module

k

is

in

resonance

choose

some

j

such

that

node

j

is

enabled

and

jwk

j

xkj

jwk

j

j

is

maximal

with

the

result

that

node

j

becomes

active

ffl

Category

Match

In

any

module

k

a

local

category

matches

a

local

input

when

node

j

is

active

module

k

is

in

resonance

and

jwk

j

xkj

jxkj

aek

-LRB-

-RRB-

If

the

match

is

unsuccessful

then

node

j

is

disabled

and

another

category

choice

is

made

If

the

match

is

successful

then

learning

takes

place

and

the

system

is

stable

until

the

input

is

removed

ffl

Learning

When

node

j

is

active

and

-LRB-

-RRB-

holds

wnew

k

j

-LRB-

wold

k

j

xk

-RRB-

-LRB-

-RRB-

wold

k

j

Properties

of

hart-j

This

section

describes

the

main

properties

of

the

hart-j

network

Stability

of

Learning

After

a

finite

number

of

input

presentations

of

an

input

set

all

the

individual

ART

modules

will

stabilise

if

the

input

is

normalised

The

following

argument

explains

why

this

is

so

Let

the

norm

of

the

input

set

be

S

Then

the

smallest

size

of

a

local

input

to

module

k

is

S

Qk

i

aei

The

learning

of

any

ART

module

with

normalised

inputs

will

stabilise

because

such

normalisation

is

a

limit

on

the

smallest

size

of

an

input

-LRB-

Carpenter

et

al.

-RRB-

This

lower

bound

on

the

size

of

inputs

has

the

same

effect

Order

of

Presentation

The

order

of

presentation

affects

the

clustering

This

follows

naturally

from

the

properties

of

the

individual

ART

modules

-LRB-

Moore

-RRB-

Direct

Access

Once

learning

has

stabilised

direct

access

to

category

prototypes

is

assured

for

each

ART

module

Given

an

input

presentation

each

module

must

wait

for

resonance

in

previous

modules

but

no

search

is

required

An

Upper

Bound

on

Effective

Vigilance

k

max

aek

Proof

In

this

model

wk

Wk

The

worst

case

is

when

xk

is

as

big

as

possible

in

fact

in

this

model

it

could

be

up

to

the

same

size

as

X

So

the

least

z

such

that

jWkj

jXj

aek

jWkj

jXj

z

for

any

X

and

any

clustering

is

aek

The

splitting

hierarchical

ART

network

-LRB-

hart-s

-RRB-

This

new

network

model

we

are

proposing

is

based

on

the

idea

of

forming

the

most

general

level

of

clusters

first

and

then

successively

splitting

those

clusters

into

more

specific

ones

Only

the

difference

between

an

input

to

a

particular

ART

module

and

the

prototype

of

the

category

of

that

input

is

propagated

to

the

next

ART

module

The

input

signal

to

the

first

module

is

the

difference

between

the

input

pattern

and

the

most

general

category

so

is

equivalent

to

the

input

pattern

itself

Only

when

such

differences

are

greater

than

a

size

determined

by

a

network

parameter

called

resolution

-LRB-

ffl

-RRB-

is

the

next

ART

module

activated

More

formally

the

hart-s

network

is

described

by

the

following

parameter

tuple

HS

fL

M

N

ae

ffl

g

where

the

meaning

of

each

parameter

is

as

follows

L

the

number

of

ART

modules

used

M

-LRB-

ML

-RRB-

the

size

of

each

and

layer

N

-LRB-

NL

-RRB-

the

size

of

each

layer

-LRB-

L

-RRB-

the

choice

parameter

of

each

ART

module

-LRB-

k

typically

k

o

-RRB-

ae

-LRB-

aeL

-RRB-

the

vigilance

level

of

each

ART

module

aek

-LSB-

-RSB-

ffl

resolution

parameter

ffl

-LSB-

-RSB-

-LRB-

L

-RRB-

the

learning

rate

of

each

ART

module

k

-LSB-

-RSB-

with

the

architectural

constraint

of

Mk

M

for

k

L.

For

further

simplicity

however

we

will

also

assume

Nk

N

k

and

k

-LRB-

i.e.

fast

learning

-RRB-

for

k

L

So

it

is

only

the

aek

vigilance

levels

that

are

allowed

to

be

different

in

each

ART

module

The

architecture

of

a

three-layer

hart-s

network

is

shown

in

figure

There

are

four

sets

of

connections

between

each

successive

module

in

a

hart-s

network

-LRB-

figure

-RRB-

An

adaptive

associative

connection

-LRB-

V

-RRB-

called

hierarchical

memory

between

the

layers

of

successive

ART

modules

provides

the

links

between

any

cluster

and

its

immediate

sub-clusters

When

a

category

is

active

in

the

first

of

a

pair

of

modules

only

those

categories

in

the

second

module

that

can

be

sub-categories

of

the

first

are

allowed

to

become

active

These

links

are

important

in

this

network

-LRB-

which

are

not

needed

in

hart-j

-RRB-

so

the

correct

hierarchies

can

be

maintained

since

the

individual

ART

modules

cluster

differences

only

The

other

inter-ART

connections

are

explained

in

figure

The

local

input

to

level

k

is

xk

xk

xk

wk

J

k

L

-LRB-

-RRB-

R

G

G

R

R

G

Pattern

Input

m-m

m-m

V

V

Figure

Architecture

of

a

three-layer

hart-s

network

Only

the

connections

between

ART

modules

are

shown

V

and

V

are

-LRB-

many-to-many

-RRB-

associative

links

between

nodes

of

neighbouring

modules

The

Ri

Gi

links

have

the

same

purpose

as

in

hart-j

The

and

one-to-one

links

are

used

to

propagate

the

difference

between

the

current

input

and

the

selected

category

prototype

to

the

next

layer

where

k

J

is

the

index

of

the

winning

node

in

module

k

Since

both

the

inputs

and

prototypes

are

binary

-LRB-

-RRB-

can

be

written

as

xk

xk

wk

J

k

L

-LRB-

-RRB-

where

denotes

the

Boolean

negation

operator

Applying

-LRB-

-RRB-

and

-LRB-

-RRB-

to

layers

k

k

recursively

-LRB-

and

ignoring

the

indices

of

winning

nodes

for

simplicity

-RRB-

for

the

local

input

to

layer

k

we

get

xk

wk

X

-LRB-

-LSB-

-LSB-

wk

-RRB-

X

Wk

k

L

-LRB-

-RRB-

where

Wk

-LRB-

or

more

precisely

Wk

J

-RRB-

is

the

global

prototype

of

category

J

at

level

k

Wk

-LSB-

-LSB-

wk

k

L

-LRB-

-RRB-

Note

that

wi

wj

i

j

k

which

follows

from

the

property

that

wi

xi

wi

and

wj

xj

wj

and

that

xj

wj

xi

wi

wj

wj

assuming

i

j

A

distinctive

feature

of

the

hart-s

network

is

that

the

local

input

xk

will

not

activate

module

k

unless

jxkj

fflS

Thus

ffl

-LRB-

or

resolution

-RRB-

controls

the

amount

of

sub-clustering

that

can

occur

Since

the

hart-s

network

develops

more

specific

hierarchical

-LRB-

or

global

-RRB-

categories

in

successive

levels

it

is

useful

to

ask

what

the

minimum

degree

of

specificity

any

level

k

category

can

have

given

a

sequence

of

training

patterns

of

size

S

and

-LRB-

local

-RRB-

vigilance

levels

aek

For

similar

reasons

to

those

in

hart-j

a

single

effective

vigilance

level

can

not

be

determined

from

the

individual

vigilance

levels

Instead

there

will

be

a

region

whose

lower

bound

-LRB-

k

min

-RRB-

will

be

of

interest

to

us

More

specifically

in

each

level

we

look

for

the

smallest

value

z

such

that

jwkj

jxkj

aek

jWkj

jXj

z

-LRB-

-RRB-

The

value

of

k

min

is

given

with

proof

in

section

Note

that

the

effective

vigilance

for

different

clusters

in

a

given

module

of

a

hart-s

network

may

be

quite

different

-LRB-

and

this

precludes

using

effective

vigilance

as

a

global

parameter

in

some

way

-RRB-

It

is

an

important

part

of

the

mechanism

since

it

allows

the

information

found

in

the

input

environment

to

determine

the

shape

of

the

resulting

hierarchy

Dynamics

of

hart-s

The

dynamics

of

the

hart-s

network

can

be

described

by

those

of

the

component

ART

modules

and

the

data/memory

connections

between

them

as

follows

For

i

M

j

h

N

and

k

L

ffl

Initialisation

The

initial

values

of

all

connection

weights

are

vk

jh

wk

ji

ffl

Layer

activity

vectors

The

layer

xk

xk

with

X

and

i

The

layer

-LRB-

xk

wk

j

when

node

j

is

active

and

module

k

is

in

resonance

xk

otherwise

The

layer

j

-LRB-

when

node

j

is

active

and

module

k

is

in

resonance

otherwise

Whenever

an

input

is

presented

module

k

is

in

resonance

and

fflS

then

any

node

j

such

that

node

h

is

active

and

vk

jh

is

enabled

and

an

initial

category

choice

is

made

ffl

Category

Choice

When

module

k

is

in

resonance

choose

j

such

that

node

j

is

enabled

and

jwk

j

xkj

jwk

j

j

-LRB-

-RRB-

is

maximal

with

the

result

that

node

j

becomes

active

ffl

Category

Match

A

category

matches

an

input

when

node

j

is

active

module

k

is

in

resonance

and

jwk

j

xkj

jxkj

aek

-LRB-

-RRB-

If

the

match

is

unsuccessful

then

node

j

is

disabled

and

another

category

choice

is

made

If

the

match

is

successful

then

learning

takes

place

and

the

module

is

stable

until

the

input

is

removed

ffl

Learning

When

node

j

is

active

and

-LRB-

-RRB-

holds

then

wnew

k

j

-LRB-

wold

k

j

xk

-RRB-

-LRB-

-RRB-

wold

k

j

-LRB-

adjustment

of

category

prototype

-RRB-

and

vnew

k

j

whereby

the

association

between

a

category

and

its

chosen

sub-category

is

created

Properties

of

hart-s

This

section

describes

the

main

properties

of

the

hart-s

network

Here

we

are

making

the

further

assumption

that

input

patterns

are

normalised

Stability

of

Learning

As

with

the

hart-j

network

all

we

need

to

do

is

show

there

is

a

lower

bound

on

the

size

of

local

inputs

for

every

ART

module

There

is

such

a

bound

since

ffl

and

a

module

will

only

be

activated

when

an

input

has

a

size

larger

than

fflS

Direct

Access

Once

learning

has

stabilised

direct

access

to

category

prototypes

is

assured

for

each

ART

module

As

in

hart-j

each

module

must

wait

for

resonance

in

the

previous

module

Order

of

Presentation

The

order

of

presentation

affects

the

clustering

Again

this

follows

naturally

from

the

properties

of

the

individual

ART

modules

A

Bound

on

the

Number

of

Levels

of

Clustering

The

most

levels

of

clustering

possible

is

B

when

the

following

equation

holds

B

minfk

kY

i

-LRB-

aei

-RRB-

fflg

-LRB-

-RRB-

Proof

Let

the

norm

of

the

inputs

be

denoted

S.

Let

the

size

of

the

largest

input

possible

to

module

k

be

Ik

S

The

smallest

prototype

formed

in

module

k

has

size

aek

Ik

For

k

Ik

Ik

-LRB-

aek

-RRB-

S

kY

i

-LRB-

aei

-RRB-

The

largest

input

to

a

possible

module

k

that

will

not

activate

it

has

size

S

ffl

So

when

Ik

ffl

module

k

can

never

be

activated

So

-LRB-

-RRB-

ffl

reduces

the

model

to

the

standard

ART

model

-LRB-

ffl

ensures

this

since

-RRB-

Consistent

Sub-clustering

If

an

input

pattern

that

is

clustered

at

a

level

k

of

a

clustering

hierarchy

is

also

clustered

at

level

k

then

all

inputs

clustered

with

it

at

level

k

will

also

be

clustered

at

level

k

if

it

exists

The

difference

in

size

between

a

local

input

and

its

local

prototype

in

a

given

module

is

the

size

of

the

input

to

the

next

module

So

it

is

sufficient

to

show

that

the

size

of

all

local

inputs

clustered

together

in

a

given

module

is

the

same

since

then

they

will

all

generate

the

same

sized

input

to

the

next

module

It

is

the

size

of

this

input

which

determines

if

the

next

module

will

become

active

and

sub-clustering

occurs

All

inputs

clustered

together

in

a

given

module

must

be

clustered

together

in

any

previous

modules

so

are

all

of

the

same

size

since

the

inputs

to

module

are

normalised

A

Lower

Bound

on

Effective

Vigilance

k

min

kY

i

-LRB-

aei

-RRB-

Proof

For

any

X

and

any

clustering

if

the

left

hand

side

of

-LRB-

-RRB-

is

true

so

must

be

the

right

hand

side

The

worst

case

is

when

the

size

of

Wk

is

as

small

as

possible

In

this

model

the

size

of

Wk

is

the

sum

of

the

sizes

of

the

local

prototypes

at

level

to

k

so

clearly

Wk

is

smallest

when

the

size

of

the

local

prototypes

in

levels

to

k

is

smallest

The

smallest

prototype

size

at

level

is

aekjXj

The

smallest

prototype

size

at

level

is

-LRB-

-RRB-

jXj

The

smallest

prototype

size

at

level

k

is

aek

Qk

i

-LRB-

aei

-RRB-

jXj

So

the

smallest

size

for

Wk

is

-LRB-

Qki

-LRB-

aei

-RRB-

-RRB-

jXj

So

the

greatest

z

such

that

the

right

hand

side

of

-LRB-

-RRB-

is

true

is

-LRB-

Qki

-LRB-

aei

-RRB-

-RRB-

Experimental

results

We

carried

out

three

sets

of

experiments

with

the

hierarchical

ART

networks

presented

here

The

primary

goal

of

each

was

To

demonstrate

the

representational

capabilities

of

hart-j

and

hart-s

networks

To

explore

the

joint

effect

of

vigilance

-LRB-

ae

-RRB-

and

resolution

-LRB-

ffl

-RRB-

of

the

hart-s

network

on

the

depth

of

the

resulting

trees

and

on

the

training

time

To

compare

the

classifications

produced

by

hart

networks

to

those

of

two

classical

hierarchical

clustering

methods

and

an

AI

conceptual

clustering

algorithm

The

details

of

the

experiments

and

the

results

are

discussed

in

the

following

sections

Representational

capabilities

of

hart

networks

In

order

to

demonstrate

that

the

hart

networks

are

capable

of

developing

class

hierarchies

we

trained

and

tested

the

networks

on

the

Zoo

machine

learning

benchmark

data

set

-LRB-

Merz

and

Murphy

-RRB-

This

data

set

contains

instances

of

animals

described

with

attributes

such

as

hair

aquatic

domestic

and

so

on

It

is

relatively

small

but

is

adequate

for

the

purposes

of

these

experiments

which

also

allows

us

to

make

general

observations

about

the

behaviour

of

these

models

that

will

guide

further

analysis

and

development

A

number

of

simulations

were

run

in

a

Unix/X

windows

environment

using

the

neural

network

simulation

program

PlaNet

-LRB-

Miyata

-RRB-

and

programs

written

in

C

language

The

binary

attributes

of

the

training

samples

were

presented

to

the

network

in

complement

coding

to

assign

significance

to

both

the

presence

and

absence

of

features

and

to

keep

the

input

vectors

normalised

-LRB-

Carpenter

et

al.

-RRB-

to

avoid

proliferation

of

ART

categories

-LRB-

Moore

-RRB-

The

only

non-binary

attribute

-LRB-

number

of

legs

-RRB-

was

presented

in

generalised

complement

coding

-LRB-

Bartfai

-RRB-

The

type

attribute

was

ignored

to

eliminate

any

bias

from

the

developed

clusterings

The

attribute

-LRB-

animal

name

-RRB-

was

simply

used

as

a

label

for

the

individual

instances

With

this

input

encoding

scheme

instances

were

presented

to

the

network

as

binary

vectors

with

the

same

norm

of

In

all

the

experiments

the

hart

networks

were

trained

off-line

on

a

training

set

containing

all

instances

of

the

zoo

database

In

each

training

session

the

network

was

considered

to

be

completely

trained

when

each

input

pattern

directly

accessed

a

matching

cluster

at

each

active

level

of

the

cluster

hierarchy

-LRB-

i.e.

no

category

search

or

further

learning

occurred

-RRB-

Of

particular

interest

are

the

effects

of

changes

in

vigilance

and

order

presentation

on

clusterings

and

the

comparative

behaviour

of

the

two

models

Figures

and

are

examples

of

hierarchies

learnt

by

three-layer

hart-j

and

hart-s

networks

respectively

The

choice

parameters

-LRB-

k

-RRB-

and

learning

rates

-LRB-

k

-RRB-

were

set

to

and

respectively

in

both

cases

In

general

we

can

observe

that

category

prototypes

are

often

duplicated

in

hart-j

as

prototypes

are

propagated

on

whether

or

not

any

clustering

of

them

is

achieved

Prototypes

in

hart-s

by

comparison

always

differ

and

furthermore

information

contained

in

one

prototype

is

not

copied

in

subcluster

prototypes

as

occurs

in

hart-j

A

difficulty

with

hart-s

is

in

predicting

the

number

of

nodes

needed

in

lower

layers

of

a

network

due

to

the

links

created

in

hierarchical

memory

during

learning

A

higher

setting

of

the

resolution

parameter

tends

to

reduce

the

number

of

such

nodes

used

to

learn

a

particular

set

of

inputs

Lastly

we

noted

that

as

with

ART

the

clusterings

learnt

by

both

hart

models

were

affected

significantly

by

a

change

in

the

order

of

input

pattern

presentation

Hair

Backbone

Toothed

No

Feathers

No

Eggs

Milk

No

Feathers

Toothed

Backbone

Breathes

Not

Venomous

Not

Domestic

Backbone

Toothed

Preditor

Aquatic

Not

Airbourne

No

Feathers

No

Hair

No

Feathers

Breathes

No

Fins

Not

Airbourne

Eggs

No

Feathers

Backbone

Breathes

Not

Venomous

No

Fins

Legs

Not

Domestic

No

Feathers

No

Milk

Not

Aquatic

Not

Toothed

No

Backbone

Breathes

No

Fins

Not

Catsize

Legs

No

Feathers

No

Milk

Not

Aquatic

Not

Toothed

No

Backbone

Breathes

No

Fins

Not

Venomous

Not

Catsize

Does

n't

Breathe

Not

Airbourne

No

Milk

No

Feathers

Eggs

No

Hair

No

Hair

No

Feathers

Eggs

No

Milk

Not

Airbourne

Preditor

No

Fins

Not

Domestic

No

Hair

No

Feathers

Eggs

No

Milk

Not

Airbourne

Does

n't

Breathe

Not

Venomous

Not

Catsize

Legs

No

Fins

Tail

Not

Venomous

Breathes

Backbone

Not

Toothed

No

Milk

Eggs

Feathers

No

Hair

No

Hair

Eggs

No

Milk

EggsNo

Feathers

Antelope

Aardvark

Bear

Boar

Dogfish

Frog

Flea

Honeybee

Crab

Octopus

Bass

Chicken

Penguin

PheasantCrow

Dove

Rhea

Duck

Flamingo

Skimmer

Skua

Gull

Sparrow

Hawk

Swan

Kiwi

Lark

Ostrich

Vulture

Wren

Parakeet

Carp

Catfish

Chub

Clam

Haddock

Herring

Sole

Pitviper

Seawasp

Slowworm

Crayfish

Lobster

Piranha

Seahorse

Starfish

Housefly

Ladybird

Moth

Scropion

Gnat

Slug

Termite

Wasp

Worm

Newt

Platypus

Toad

Tortoise

Tuatara

Dolphin

Pike

Porpoise

Seasnake

Stingray

Tuna

Girl

Goat

Gorilla

Hamster

Hare

Leopard

Lion

Lynx

Mole

Opossum

Oryx

Buffalo

Calf

Cavy

Cheetah

Deer

Elephant

Fruitbat

Giraffe

Pony

Puma

Polecat

Pussycat

Raccoon

Reindeer

Squirrel

Vampire

Vole

Mongoose

Wolf

Wallaby

Mink

Seal

Sealion

Figure

A

cluster

hierarchy

learnt

from

the

Zoo

database

by

a

three

layer

hart-j

network

The

vigilance

levels

were

and

Cluster

prototypes

specify

all

the

features

required

for

cluster

membership

and

thus

always

increase

in

size

as

the

hierarchy

is

descended

The

root

of

the

tree

is

only

shown

here

for

clarity

but

is

not

actually

represented

in

the

network

In

this

example

ART

modules

and

developed

and

categories

respectively

The

tree

structure

emerges

as

input

patterns

are

presented

and

the

corresponding

categories

are

activated

at

each

level

The

instances

belonging

to

each

leaf

are

shown

here

as

the

lowest

layer

-LRB-

which

could

also

have

been

learned

by

the

network

with

an

ART

module

inserted

before

the

first

one

with

a

vigilance

level

of

-RRB-

Flea

Gnat

Honeybee

Housefly

Ladybird

Moth

Slug

Termite

Wasp

Worm

Bass

Carp

Catfish

Chub

Dogfish

Haddock

Herring

Pike

Newt

Tuna

Tuatara

Stingray

Sole

Slowworm

Seahorse

Pitviper

Piranha

Toad

Starfish

Seawasp

Octopus

Lobster

Frog

Crayfish

Crab

Clam

No

Fins

Chicken

Crow

Dove

Duck

Flamingo

Gull

Hawk

Kiwi

Lark

Ostrich

Parakeet

Penguin

Pheasant

Rhea

Skimmer

Skua

Sparrow

Swan

Tortoise

Vulture

Toothed

No

Tail

No

Backbone

Not

Aquatic

No

Feathers

No

Tail

Tail

Backbone

Tail

Backbone

Wren

Scropion

Seasnake

Hair

Not

Airbourne

Not

Aquatic

No

Fins

Cavy

Calf

Girl

Goat

Hamster

Pony

Reindeer

Preditor

Aquatic

Fins

Dolphin

Porpoise

Seal

Sealion

Aardvark

Antelope

Bear

Boar

Buffalo

Cheetah

Deer

Elephant

Fruitbat

Giraffe

Gorilla

Mink

Mole

Mongoose

Opossum

Oryx

Polecat

Squirrel

Raccoon

Pussycat

Puma

Platypus

No

Fins

Hair

Domestic

Not

Domestic

Not

Airbourne

No

Feathers

No

Fins

Breathes

No

Milk

Not

VenemousBreathes

Backbone

Milk

No

Feathers

Not

Toothed

Not

Domestic

Leopard

Hare

Lion

Lynx

Wallaby

Wolf

Vampire

Vole

Figure

A

cluster

hierarchy

learnt

from

the

Zoo

database

by

a

three

layer

hart-s

network

The

resolution

-LRB-

ffl

-RRB-

was

and

all

three

vigilance

levels

were

set

to

Here

the

hierarchy

is

explicitly

encoded

in

the

network

via

the

vk

j

associative

links

Any

particular

feature

is

only

used

to

specify

a

particular

branch

of

the

tree

The

usage

of

the

root

and

leaves

is

the

same

is

in

figure

Depth

Resolution

Vigilance

Figure

The

average

depth

of

hart-s

hierarchies

-LRB-

vertical

axis

-RRB-

with

respect

to

vigilance

and

resolution

-LRB-

ae

and

ffl

-RRB-

after

training

with

the

Zoo

database

Each

point

on

the

graph

is

the

average

of

clusterings

with

different

orders

of

input

pattern

presentation

The

flat

area

indicating

only

one

level

of

clustering

is

formed

where

the

network

is

constrained

to

behave

as

a

single

ART

module

Lower

settings

of

vigilance

tend

to

result

in

smaller

prototypes

in

any

ART

modules

and

thus

more

levels

of

clustering

Lower

settings

of

resolution

subcluster

smaller

differences

with

the

same

effect

The

joint

effect

of

vigilance

and

resolution

This

set

of

experiments

explored

the

combined

effect

of

vigilance

-LRB-

ae

-RRB-

and

resolution

-LRB-

ffl

-RRB-

in

hart-s

on

the

depth

of

the

resulting

hierarchy

and

on

required

training

time

The

data

set

training

conditions

and

default

parameter

settings

were

identical

to

those

used

in

the

previous

experiments

Because

of

the

significant

effect

of

the

order

of

input

pattern

presentation

each

set

of

simulations

consisted

of

runs

each

with

a

different

ordering

of

inputs

and

the

measurements

averaged

out

The

average

deviations

of

these

measurements

indicate

they

should

be

quite

reliable

guides

to

the

expected

behaviour

of

hart-s

The

depth

of

a

hierarchy

was

taken

as

the

deepest

subcluster

and

training

time

was

taken

as

the

number

of

presentations

of

the

set

of

input

patterns

that

caused

any

change

of

local

category

prototypes

Figures

and

show

the

average

depth

and

training

time

-LRB-

in

epochs

-RRB-

respectively

while

each

of

resolution

and

vigilance

which

was

the

same

for

each

module

was

varied

from

to

in

steps

of

The

relationship

expressed

in

equation

-LRB-

-RRB-

is

clearly

reflected

in

the

first

figure

and

in

the

second

it

is

clear

that

the

training

time

required

to

create

a

cluster

hierarchy

increases

relatively

slowly

with

increasing

tree

depth

Perhaps

the

symmetry

of

graphs

suggest

a

redundant

parameter

However

clusterings

of

the

same

depth

and

learnt

in

the

same

number

of

epochs

but

with

significantly

different

combinations

of

ffl

and

ae

are

usually

very

different

Epochs

Resolution

Vigilance

Figure

The

average

training

time

for

hart-s

hierarchies

with

respect

to

vigilance

and

resolution

-LRB-

ae

and

ffl

-RRB-

after

training

on

the

Zoo

database

A

unit

on

the

vertical

axis

corresponds

to

one

epoch

in

which

the

entire

set

of

input

patterns

is

presented

Each

point

on

the

graph

is

the

average

of

clusterings

with

different

orders

of

input

pattern

presentation

these

being

the

same

ones

as

used

in

figure

Comparison

to

other

clusterings

methods

The

primary

motivation

for

designing

these

hierarchical

ART

networks

was

to

have

a

modular

network

that

is

capable

of

fast

and

stable

incremental

learning

of

hierarchical

clusterings

of

arbitrary

sequences

of

input

patterns

by

self-organisation

Having

achieved

this

goal

we

can

now

take

a

look

at

these

networks

from

a

different

angle

and

view

them

simply

as

implementations

of

incremental

clustering

algorithms

The

question

that

arises

then

is

how

these

algorithms

as

such

compare

to

other

existing

clustering

methods

We

attempted

to

find

an

answer

to

this

with

this

set

of

experiments

We

compared

the

clusterings

gained

by

the

hart-j

and

hart-s

networks

to

standard

statistical

clustering

methods

and

an

incremental

clustering

machine

learning

method

The

following

two

data

sets

from

the

UCI

Repository

-LRB-

Merz

and

Murphy

-RRB-

were

used

ffl

Soybean

this

data

set

contains

instances

described

with

nominal

attributes

-LRB-

and

class

information

which

was

ignored

here

-RRB-

The

attributes

were

encoded

in

generalised

complement

coding

-LRB-

see

above

-RRB-

with

missing

attribute

values

represented

on

a

separate

bit

yielding

binary

vectors

of

norm

ffl

Mushroom

this

data

set

contains

randomly

selected

examples

from

the

original

Mushroom

set

found

in

the

repository

Each

instance

is

described

with

nominal

features

-LRB-

and

class

information

which

was

ignored

here

-RRB-

Again

attributes

were

encoded

in

generalised

complement

coding

giving

binary

vectors

-LRB-

of

norm

-RRB-

The

two

statistical

hierarchical

clustering

methods

that

were

used

for

comparisons

are

compact

clustering

-LRB-

also

known

as

complete

linkage

-RRB-

and

group

average

-LRB-

Gordon

-RRB-

Both

of

them

are

agglomerative

methods

In

such

methods

the

initial

configuration

involves

treating

each

instance

as

a

cluster

of

size

one

At

any

subsequent

stage

two

clusters

are

amalgamated

into

one

using

the

two

clusters

judged

closest

by

some

suitable

distance

measure

Clustering

ceases

when

all

observations

are

combined

into

one

cluster

In

case

of

compact

clustering

the

distance

between

two

clusters

is

taken

as

the

maximum

of

the

distances

between

any

point

of

the

first

cluster

and

any

point

in

the

second

cluster

Group

average

clustering

is

similar

to

compact

clustering

differing

only

in

the

criterion

by

which

extant

clusters

are

to

be

fused

at

any

given

stage

Here

the

criterion

is

the

average

of

the

distances

between

points

in

the

two

clusters

The

built-in

library

functions

of

the

S-PLUS

statistical

package

were

used

here

While

the

above

clustering

methods

are

non-incremental

-LRB-

i.e.

the

whole

data

set

has

to

be

available

for

the

computation

-RRB-

several

incremental

machine

learning

algorithms

have

been

proposed

for

unsupervised

concept

formation

One

such

method

is

called

Froggit

Froggit

is

an

incremental

top-down

hierarchical

clustering

algorithm

derived

from

COBWEB

-LRB-

Fisher

-RRB-

and

CLASSIT

-LRB-

Gennari

et

al.

-RRB-

Like

COBWEB

it

processes

new

instances

one

at

a

time

incorporating

them

into

a

class

hierarchy

and

attempting

to

incrementally

optimise

the

hierarchy

Froggit

differs

from

COBWEB

and

CLASSIT

in

three

significant

ways

First

Froggit

uses

an

information

theoretic

measure

of

the

goodness

of

a

partition

in

its

optimisation

search

This

measure

allows

Froggit

to

handle

both

numeric

and

discrete

attributes

in

a

well-founded

way

The

measure

is

based

on

a

minimum

message

length

approach

-LRB-

Wallace

and

Boulton

-RRB-

but

is

not

completely

rigorous

Second

the

search

strategy

in

the

optimisation

phase

is

much

richer

than

COBWEB

's

It

uses

a

beam

hill-climbing

search

and

are

two

versions

of

this

data

set

in

the

UCI

repository

soybean-small

and

soybean-large

We

used

the

latter

is

exactly

the

same

encoding

that

was

used

in

-LRB-

Carpenter

et

al.

-RRB-

earlier

version

of

Froggit

called

DySect

has

been

applied

successfully

to

data

analysis

-LRB-

Dawkins

et

al.

-RRB-

Compact

Group

Froggit

Nc

Pr

Phs

Phm

Phs

Phm

Phs

Phm

Table

This

table

shows

estimates

of

agreement

probabilities

between

hart-j

and

three

other

clustering

methods

on

the

Soybean

and

Mushroom

data

sets

-LRB-

Phs

and

Phm

columns

respectively

for

each

method

-RRB-

The

's

indicate

that

Froggit

could

not

find

exactly

Nc

categories

in

which

case

the

probabilities

were

taken

from

a

partition

having

the

closest

number

of

categories

a

wider

range

of

hierarchy

modification

operators

than

COBWEB

These

allow

it

to

smooth

the

search

space

and

avoid

some

of

the

local

minima

that

trap

COBWEB

Third

Froggit

separates

the

optimisation

from

the

incorporation

which

not

only

improves

the

effectiveness

of

the

search

but

also

allows

a

wider

range

of

incorporation

strategies

-LRB-

Andreae

-RRB-

Since

the

class

representation

of

all

of

the

above

methods

-LRB-

including

hart

-RRB-

are

markedly

different

a

reasonable

way

to

compare

the

methods

is

to

look

at

their

partitionings

at

the

instance

level

-LRB-

instead

of

comparing

their

class

descriptions

-RRB-

The

developed

clusterings

of

the

hart

networks

however

are

affected

by

the

order

of

input

presentation

Therefore

instead

of

determining

a

single

similarity

value

we

can

estimate

the

probability

with

which

categories

of

hart

networks

and

other

methods

agree

More

precisely

given

the

same

number

of

categories

found

we

estimate

Phd

-LRB-

cij

-RRB-

i

j

i

j

D

where

D

is

the

size

of

the

data

set

d

cij

and

hij

if

instances

i

and

j

are

assigned

to

the

same

class

by

a

clustering

algorithm

and

a

hart

network

in

a

given

training

run

respectively

In

other

words

Phd

is

the

probability

that

if

instances

i

and

j

are

found

to

be

in

the

same

category

by

a

hart

network

they

are

also

found

to

be

in

the

same

category

by

a

clustering

algorithm

The

best

case

is

when

Phd

i.e.

the

two

methods

agree

of

the

time

The

worst

case

is

complete

randomness

or

non-correlation

between

the

methods

However

even

in

this

case

there

is

a

finite

probability

that

the

random

selection

of

categories

will

agree

with

any

clustering

method

In

case

of

complete

random

categories

this

probability

-LRB-

Pr

-RRB-

is

where

Nc

is

the

number

of

classes

in

the

given

partition

So

clearly

Pr

Phd

Table

and

show

the

estimated

Phd

probabilities

for

the

hart-j

and

hart-s

networks

respectively

The

number

of

classes

-LRB-

Nc

-RRB-

were

gained

differently

in

each

method

For

the

statistical

clustering

methods

-LRB-

Compact

and

Group

-RRB-

a

cut

of

the

tree

was

taken

at

the

desired

level

For

Froggit

one

of

its

parameters

which

controls

the

specificity

-LRB-

and

therefore

the

number

-RRB-

of

top-level

classes

in

the

hierarchy

was

varied

until

the

desired

number

was

produced

In

hart-j

different

number

of

classes

were

gained

by

simply

observing

the

partitionings

of

different

modules

in

the

ART

cascade

In

hart-s

the

appropriate

cut

of

the

tree

was

made

through

the

use

of

the

resolution

parameter

-LRB-

ffl

-RRB-

in

a

trained

network

-LRB-

varying

it

between

and

its

value

at

training

-RRB-

which

typically

resulted

in

cuts

spanning

multiple

ART

modules

First

the

results

show

that

the

hart

networks

produce

clusterings

that

are

correlated

-LRB-

at

least

according

to

our

measure

-RRB-

to

the

other

methods

at

better

-LRB-

sometimes

much

better

-RRB-

Compact

Group

Froggit

Nc

Pr

Phs

Phm

Phs

Phm

Phs

Phm

Table

This

table

shows

estimates

of

agreement

probabilities

between

hart-s

and

three

other

clustering

methods

on

the

Soybean

and

Mushroom

data

sets

For

more

explanation

refer

to

table

Compact

Group

Froggit

ART

hart-j

hart-s

Froggit

Table

The

comparison

of

three

ART-based

methods

and

Froggit

against

the

three

base

ones

used

above

The

numbers

are

the

same

Phd

probabilities

used

in

tables

and

Here

we

show

results

for

classes

-LRB-

i.e.

Pr

-RRB-

learned

from

the

Mushroom

database

than

chance

level

It

can

also

be

seen

clearly

that

the

numbers

for

hart-j

and

hart-s

are

very

similar

suggesting

that

even

though

the

architectures

and

developed

class

representations

are

different

they

produce

similar

clusterings

The

similarity

between

hart

and

the

other

is

noticeably

higher

for

the

Mushroom

data

set

The

Group

method

and

Froggit

are

apparently

more

similar

to

hart

than

the

Compact

method

For

the

two

classical

clustering

methods

the

level

of

agreement

tends

to

increase

as

Nc

decreases

We

also

ran

some

experiments

to

gauge

the

similarity

of

clusterings

of

single

ART

networks

to

those

of

hart-j

hart-s

and

Froggit

-LRB-

against

the

three

base

methods

used

in

the

tables

above

-RRB-

as

well

as

to

see

how

Froggit

compares

to

the

two

statistical

clustering

methods

Table

shows

the

results

on

the

Mushroom

database

when

the

number

of

classes

was

A

very

interesting

result

is

that

the

hart

and

ART

networks

-LRB-

and

Froggit

-RRB-

show

remarkably

similar

levels

of

agreement

with

the

compact

and

group

average

methods

This

seems

to

suggest

that

the

hierarchical

ART

networks

offer

their

added

representational

power

without

any

apparent

change

in

their

clustering

behaviour

Also

the

hart

networks

and

Froggit

appear

to

be

closer

to

each

other

than

to

the

classical

methods

while

they

are

also

slightly

closer

to

Froggit

than

the

ART

network

Recall

however

that

the

aim

has

been

to

provide

a

modular

neural

network

model

that

inherits

the

clustering

properties

of

some

kind

of

ART

type

components

and

integrates

them

to

perform

the

task

of

discovering

a

set

of

useful

partitions

and

a

hierarchical

relationship

between

them

that

is

both

consistent

and

usable

-LRB-

rather

than

to

find

a

hierarchical

clustering

method

better

than

for

example

Froggit

-RRB-

Discussion

In

the

following

we

discuss

other

neural

network

models

that

we

consider

to

be

more

closely

related

to

the

hierarchical

ART

networks

introduced

here

We

shall

focus

on

identifying

similarities

and

differences

Lampinen

and

Oja

proposed

a

model

in

which

a

hierarchy

of

self-organising

maps

-LRB-

SOMs

see

-LRB-

Kohonen

-RRB-

-RRB-

was

used

for

visual

feature

extraction

-LRB-

Lampinen

and

Oja

-RRB-

In

a

MSOM

network

there

is

a

separate

input

vector

for

each

SOM

network

in

the

first

layer

The

input

to

the

second

layer

is

a

vector

with

the

index

of

the

winning

node

in

each

SOM

network

of

the

first

layer

The

authors

admit

this

does

not

bear

any

resemblance

to

real

networks

but

provides

a

useful

and

effective

method

for

feature

extraction

in

their

application

This

hierarchy

is

rather

different

from

that

of

a

hart

network

both

in

what

input

each

layer

receives

and

how

the

layers

are

interconnected

In

particular

the

hierarchy

in

a

hart

is

not

encoded

explicitly

in

the

network

architecture

which

emerges

-LRB-

via

learned

links

-RRB-

as

the

network

undergoes

training

Li

et

al.

proposed

a

neural

tree

architecture

for

hierarchical

classification

and

vector

quantisation

-LRB-

Li

et

al.

-RRB-

It

implements

a

splitting

-LRB-

or

divisive

-RRB-

mechanism

by

selforganisation

like

hart-s

However

in

a

neural

tree

each

node

in

every

layer

receives

the

same

-LRB-

global

-RRB-

input

pattern

and

the

layers

and

nodes

are

selected

during

learning

and

recall

using

a

global

control

algorithm

In

hart

only

the

lowest

layer

-LRB-

or

module

-RRB-

receives

the

original

input

pattern

and

its

control

is

completely

autonomous

and

emerges

as

a

result

of

interaction

between

layers

and

nodes

using

only

local

operations

After

training

the

neural

tree

is

optimised

by

pruning

and

splitting

taking

ideas

from

machine

learning

-LRB-

e.g.

CART

-LRB-

Breiman

et

al.

-RRB-

-RRB-

The

neural

tree

was

applied

to

data

compression

and

speech

processing

Despite

their

usefulness

in

practical

applications

the

above

networks

would

not

be

suitable

in

autonomous

learning

tasks

because

of

slow

learning

and

instability

against

changes

in

the

environment

-LRB-

see

for

example

-LRB-

Moore

-RRB-

-RRB-

Partly

motivated

by

these

problems

researchers

have

looked

at

hierarchical

ART

systems

in

recent

years

The

architecture

-LRB-

Carpenter

and

Grossberg

-RRB-

can

be

considered

as

the

first

example

of

hierarchical

ART

networks

Carpenter

et

al.

define

two

modular

ART-based

models

-LRB-

ART

bidirectional

hierarchy

and

ART

cascade

-RRB-

for

learning

hierarchical

clusterings

in

which

there

is

a

close

coupling

between

each

ART

module

mediated

by

special

gate

control

signals

and

each

module

learns

information

that

is

already

processed

by

previous

layers

This

mechanism

is

very

similar

to

hart

except

for

one

main

difference

in

each

component

network

performs

contrast

enhancement

-LRB-

Carpenter

and

Grossberg

-RRB-

in

the

layer

instead

of

selecting

a

winner

The

output

pattern

of

the

layer

is

then

used

as

input

to

the

next

ART

layer

In

contrast

information

that

is

passed

on

to

successive

layers

in

a

hart

network

is

based

on

prototypes

-LRB-

either

directly

or

as

a

difference

vector

-RRB-

rather

than

distances

of

inputs

from

the

category

prototypes

-LRB-

like

in

-RRB-

Also

because

in

each

layer

is

set

to

produce

partially

compressed

codes

via

nonlinear

differential

equations

calculating

the

output

pattern

is

computationally

more

expensive

than

in

hart

where

categories

are

accessed

directly

in

a

trained

network

resulting

in

faster

recall

of

learned

prototypes

An

interesting

and

seemingly

closely

related

ART-based

hierarchical

model

is

the

ART

tree

-LRB-

Caudell

et

al.

-RRB-

that

was

designed

to

be

embedded

in

a

Neural

Information

Retrieval

System

-LRB-

NIRS

-RRB-

to

facilitate

the

intelligent

and

fast

retrieval

of

engineering

design

The

ART

tree

is

a

collection

of

macrocircuits

connected

in

a

tree

topology

Layers

in

the

tree

are

trained

successively

by

first

breaking

up

the

set

of

designs

into

a

few

general

categories

and

then

training

the

modules

in

lower

layers

on

only

those

patterns

that

belong

to

their

respective

parent

class

The

main

differences

between

the

ART

tree

and

hart

are

that

in

an

ART

tree

-LRB-

i

-RRB-

each

macrocircuit

is

allowed

to

see

the

entire

-LRB-

global

-RRB-

input

pattern

-LRB-

ii

-RRB-

the

hierarchy

is

explicitly

encoded

in

the

network

prior

to

learning

in

the

form

of

a

tree

and

-LRB-

iii

-RRB-

learning

and

recall

are

under

strict

global

control

as

part

of

the

NIRS

CAD

system

Also

the

ART

tree

however

generic

in

principle

is

application

specific

-LRB-

with

different

layers

being

selective

to

given

features

only

-RRB-

whereas

hart

does

not

make

any

assumptions

about

the

relative

importance

of

input

features

Soon

and

Tan

proposed

the

concept

hierarchy

network

-LRB-

CHN

in

-LRB-

Soon

and

Tan

-RRB-

-RRB-

as

an

ART-based

memory

model

that

forms

concept

hierarchy

by

learning

sample

relations

between

concepts

All

concepts

are

represented

in

a

concept

layer

Relations

between

a

concept

and

its

defining

lower

level

concepts

are

chunked

as

cognitive

codes

represented

in

a

coding

layer

This

network

is

a

good

example

of

how

ART

networks

can

be

combined

to

learn

more

complex

tasks

which

individual

components

can

not

do

by

themselves

However

even

though

the

network

is

modular

and

learns

a

hierarchy

these

can

not

be

seen

as

the

main

design

principle

in

either

the

architecture

or

the

learning

mechanism

It

shows

up

for

example

in

how

the

CHN

is

limited

to

two-level

hierarchies

whereas

in

hart

the

number

of

levels

is

variable

and

depends

on

the

training

set

and

the

vigilance

level

of

component

ART

modules

The

first

of

our

ART-based

models

-LRB-

smart

for

Self-consistent

Modular

ART

-RRB-

was

introduced

in

-LRB-

Bartfai

-RRB-

in

which

layers

in

a

hierarchy

were

connected

through

an

associative

layer

much

the

same

way

as

the

two

ART

modules

are

connected

in

an

ARTMAP

network

-LRB-

Carpenter

et

al.

-RRB-

In

the

smart

network

each

module

learned

to

cluster

the

original

inputs

at

different

levels

of

vigilance

and

self-consistency

was

maintained

via

ARTMAP

's

match

tracking

process

This

is

a

rather

different

alternative

approach

to

ART

based

hierarchical

clustering

resulting

in

markedly

different

developed

hierarchies

-LRB-

Bartfai

-RRB-

More

recently

Ishihara

et

al.

introduced

arboART

-LRB-

Ishihara

et

al.

-RRB-

an

ART

based

architecture

which

is

identical

in

basic

principles

to

our

earlier

model

-LRB-

hart-j

-RRB-

In

other

words

both

arboART

and

hart-j

develop

class

hierarchies

by

clustering

the

prototypes

developed

at

the

lower

level

The

only

significant

difference

is

that

arboART

is

based

on

the

network

-LRB-

Ishihara

et

al.

-RRB-

which

is

an

adaptation

of

to

analogue

patterns

for

small

sample

size

The

clustering

performance

of

arboART

was

found

to

be

comparable

to

that

of

a

the

centroid

method

in

statistical

clustering

The

existence

of

arboART

and

hart-j

demonstrates

well

the

advantage

of

such

architecture

where

different

types

of

ART

networks

can

be

plugged

in

to

achieve

certain

goals

according

to

different

criteria

We

also

note

that

the

hart

networks

possess

certain

features

of

self-similar

fractal-like

architectures

which

are

also

advantages

of

modular

networks

-LRB-

Happel

and

Murre

-RRB-

Namely

ART

modules

in

a

hart-j

or

hart-s

network

interact

via

modifiable

links

as

well

as

fixed

-LRB-

non-specific

-RRB-

control

signals

similar

to

the

way

layers

and

individual

nodes

interact

within

a

single

ART

network

This

self-similarity

can

be

carried

even

further

in

the

hart-s

network

in

which

training

starts

with

a

pool

of

uncommitted

ART

modules

that

are

later

allocated

as

needed

depending

on

the

training

set

and

a

global

parameter

-LRB-

resolution

in

section

-RRB-

much

like

individual

nodes

become

committed

in

a

single

ART

module

depending

on

a

within-ART

global

parameter

-LRB-

vigilance

-RRB-

This

opens

up

a

number

of

possibilities

for

creating

more

sophisticated

interactions

between

ART

layers

to

direct

its

search

in

the

hierarchy

and

enhance

its

performance

-LRB-

Another

good

example

for

combining

modularity

and

hierarchy

in

a

self-similar

structure

is

-LRB-

Jordan

and

Jacobs

-RRB-

-RRB-

As

a

final

note

the

hart-s

network

essentially

implements

the

same

kind

of

hierarchical

search

and

recall

that

was

discussed

in

-LRB-

Ambros-Ingerson

et

al.

-RRB-

However

the

recall

there

was

achieved

by

temporal

sampling

of

the

same

input

signal

-LRB-

dog

's

sniffing

-RRB-

until

the

network

found

the

most

specific

category

that

described

the

input

Because

of

the

reliance

of

repetitive

sampling

that

network

was

able

to

represent

the

hierarchy

in

a

single

layer

-LRB-

with

appropriate

settings

of

network

dynamics

-RRB-

So

we

can

view

the

hart-s

network

as

having

explicit

spatial

representation

of

the

hierarchy

which

compensates

for

the

lack

of

repetitive

sampling

Conclusions

This

paper

proposed

a

modular

neural

network

architecture

-LRB-

hart-s

-RRB-

that

is

capable

of

stable

and

fast

learning

of

hierarchical

clusterings

of

input

sequences

by

dividing

initially

large

classes

into

smaller

more

specific

ones

The

network

is

a

cascade

of

Adaptive

Resonance

Theory

modules

each

clustering

the

differences

between

the

input

to

the

previous

module

and

its

selected

category

prototype

The

class

hierarchy

is

learned

via

associative

links

between

layers

of

neighbouring

modules

The

depth

of

the

hierarchy

can

be

controlled

with

a

global

parameter

called

resolution

We

have

shown

that

the

network

like

the

joining

Hierarchical

ART

we

also

discussed

here

retains

the

useful

properties

of

the

component

ART

networks

The

experiments

demonstrated

the

representational

and

learning

capabilities

of

both

the

hart-j

and

hart-s

networks

and

also

suggested

that

the

developed

clusterings

might

be

closely

related

those

produced

by

some

classical

and/or

machine

learning

clustering

algorithms

We

believe

investigating

such

similarities

-LRB-

and

differences

too

-RRB-

further

could

provide

us

with

more

insights

into

the

advantages

and

drawbacks

of

using

ART-based

modular

networks

in

real-world

problems

This

may

also

allow

us

to

address

issues

such

as

extracting

rules

from

trained

hart

networks

and/or

optimising

the

developed

hierarchies

Along

the

latter

line

it

may

be

useful

to

view

hart

networks

-LRB-

especially

in

fast

learning

mode

-RRB-

as

implementing

the

first

sorting

stage

in

a

multi-stage

clustering

mechanism

-LRB-

Fisher

-RRB-

We

also

note

that

as

a

consequence

of

the

modular

architecture

of

the

hart-s

and

hart-j

networks

it

is

relatively

simple

to

modify

them

to

achieve

slightly

different

functionalities

For

example

the

network

can

learn

analogue

patterns

by

replacing

the

components

with

-LRB-

Carpenter

and

Grossberg

-RRB-

or

Fuzzy

ART

-LRB-

Carpenter

et

al.

-RRB-

modules

A

probabilistic

hierarchical

network

based

on

Bayesian

decision

principles

can

be

gained

by

using

Gaussian

ART

-LRB-

Williamson

-RRB-

modules

Finally

we

believe

the

hierarchical

networks

introduced

here

are

not

only

good

examples

of

modularity

but

can

also

suggest

possible

mechanisms

for

learning

complex

recognition

tasks

in

real

and

artificial

neural

networks

We

also

believe

this

model

is

worth

investigating

further

which

could

lead

us

to

a

refined

model

capturing

some

of

the

more

complex

phenomena

that

occur

in

both

artificial

and

biological

systems

Acknowledgements

This

work

was

in

part

supported

by

Victoria

University

of

Wellington

-LRB-

Research

Grant

No.RGNT

-RRB-

as

well

as

the

Department

of

Computer

Science

We

wish

to

acknowledge

the

programming

support

of

David

Gilligan

We

also

thank

the

reviewers

for

their

comments

which

helped

us

improve

the

paper

References

Ambros-Ingerson

J.

Granger

R.

and

Lynch

G.

-LRB-

-RRB-

Simulation

of

paleocortex

performs

hierarchical

clustering

Science

-LCB-

Andreae

P.

-LRB-

-RRB-

Froggit

's

goodness

measure

and

search

strategy

Personal

communication

Bartfai

G.

-LRB-

-RRB-

Hierarchical

Clustering

with

ART

Neural

Networks

In

Proceedings

of

the

IEEE

International

Conference

on

Neural

Networks

volume

pages

-LCB-

IEEE

Press

Bartfai

G.

-LRB-

-RRB-

A

Comparison

of

Two

ART-based

Neural

Networks

for

Hierarchical

Clustering

In

Kasabov

K.

and

Coghill

G.

editors

ANNES

The

Second

New

Zealand

International

Two-Stream

Conference

on

Artificial

Neural

Networks

and

Expert

Systems

pages

-LCB-

IEEE

Computer

Society

Press

Bartfai

G.

-LRB-

-RRB-

An

ART-based

Modular

Architecture

for

Learning

Hierarchical

Clusterings

Technical

Report

Department

of

Computer

Science

Victoria

University

of

Wellington

New

Zealand

Accepted

for

publication

in

Neurocomputing

Breiman

L.

Friedman

J.

Plshen

R.

and

Sonte

C.

-LRB-

-RRB-

Classification

and

Regression

Trees

Wadsworth

and

Brooks

Monterey

CA

Carpenter

G.

and

Grossberg

S.

-LRB-

-RRB-

A

Massively

Parallel

Architecture

for

a

SelfOrganizing

Neural

Pattern

Recognition

Machine

Computer

Vision

Graphics

and

Image

Processing

-LCB-

Carpenter

G.

and

Grossberg

S.

-LRB-

-RRB-

Self-Organizing

of

Stable

Category

Recognition

Codes

for

Analog

Input

Patterns

Applied

Optics

-LRB-

-RRB-

-LCB-

Carpenter

G.

and

Grossberg

S.

-LRB-

-RRB-

Hierarchical

Search

Using

Chemical

Transmitters

in

Self-Organizing

Pattern

Recognition

Architectures

Neural

Networks

-LCB-

Carpenter

G.

Grossberg

S.

and

Reynolds

J.

-LRB-

-RRB-

ARTMAP

Supervised

Real-Time

Learning

and

Classification

of

Nonstationary

Data

by

a

Self-Organizing

Neural

Network

Neural

Networks

-LCB-

Carpenter

G.

Grossberg

S.

and

Rosen

D.

-LRB-

-RRB-

Fuzzy

ART

Fast

Stable

Learning

and

Categorization

of

Analog

Patterns

by

an

Adaptive

Resonance

System

Neural

Networks

-LCB-

Carpenter

G.

A.

and

Grossberg

S.

-LRB-

-RRB-

The

ART

of

Adaptive

Pattern

Recognition

by

a

Self-Organizing

Neural

Network

IEEE

Computer

-LRB-

-RRB-

-LCB-

Caudell

T.

P.

Smith

S.

D.

G.

Escobedo

R.

and

Anderson

M.

-LRB-

-RRB-

NIRS

Large

Scale

Neural

Architectures

for

Engineering

Design

Retrieval

Neural

Networks

-LRB-

-RRB-

Dawkins

B.

P.

Andreae

P.

M.

and

O'Connor

P.

M.

-LRB-

-RRB-

Analysis

of

olympic

heptathlon

data

Journal

of

the

American

Statistical

Association

-LRB-

-RRB-

-LCB-

Fisher

D.

-LRB-

-RRB-

Knowledge

Acquisition

Via

Incremental

Conceptual

Clustering

Machine

Learning

-LRB-

-RRB-

-LCB-

Fisher

D.

-LRB-

-RRB-

Iterative

Optimization

and

Simplification

of

Hierarchical

Clusterings

Journal

of

Artificial

Intelligence

Research

-LCB-

Gennari

J.

Langley

P.

and

Fisher

D.

-LRB-

-RRB-

Models

of

Incremental

Concept

Formation

Artificial

Intelligence

-LCB-

Gordon

A.

D.

-LRB-

-RRB-

Classification

Methods

for

the

Exploratory

Analysis

of

Multivariate

Data

London

Chapman

and

Hall

Happel

B.

and

Murre

J.

-LRB-

-RRB-

Design

and

evolution

of

modular

neural

network

architectures

Neural

Networks

-LRB-

-RRB-

-LCB-

Hartigan

J.

A.

-LRB-

-RRB-

Clustering

Algorithms

John

Wiley

Sons

Inc.

Hecht-Nielsen

R.

-LRB-

-RRB-

Counterpropagation

networks

Applied

Optics

-LRB-

-RRB-

-LCB-

Hrycej

T.

-LRB-

-RRB-

Modular

Learning

in

Neural

Networks

A

Modularized

Approach

to

Neural

Network

Classification

Sixth-Generation

Computer

Technology

Series

John

Wiley

Sons

Inc.

Ishihara

S.

Ishihara

K.

Nagamachi

M.

and

Matsubara

Y.

-LRB-

-RRB-

ART

for

Kansei

engineering

expert

system

In

Proceedings

of

International

Joint

Conference

on

Neural

Networks

pages

-LCB-

Ishihara

S.

Ishihara

K.

Nagamachi

M.

and

Matsubara

Y.

-LRB-

-RRB-

arboART

ART

based

hierarchical

clustering

and

its

application

to

questionnaire

data

analysis

In

Proceedings

of

the

IEEE

International

Conference

on

Neural

Networks

volume

pages

-LCB-

IEEE

Press

Jordan

M.

I.

and

Jacobs

R.

A.

-LRB-

-RRB-

Hierarchies

of

Adaptive

Experts

In

Moody

J.

E.

Hanson

S.

J.

and

Lippmann

R.

P.

editors

Advances

in

Neural

Information

Processing

Systems

pages

-LCB-

Morgan

Kaufmann

San

Mateo

CA

Jordan

M.

I.

and

Jacobs

R.

A.

-LRB-

-RRB-

Hierarchical

Mixtures

of

Experts

and

the

EM

algorithm

Technical

Report

MIT

Computational

Cognitive

Science

MIT

Cambridge

MA

Kohonen

T.

-LRB-

-RRB-

Self-organized

formation

of

topologically

correct

feature

maps

Biological

Cybernetics

-LCB-

Lampinen

J.

and

Oja

E.

-LRB-

-RRB-

Clustering

properties

of

hierarchical

self-organizing

maps

Journal

of

Mathematical

Imaging

and

Vision

-LCB-

LeCun

Y.

Boser

B.

Denker

J.

Henderson

D.

Howard

R.

Hubbard

W.

and

Jackel

L.

-LRB-

-RRB-

Back-propagation

applied

to

handwritten

zip

code

recognition

Neural

Computation

-LCB-

Li

T.

Fang

L.

and

Li

K.

Q.-Q

-LRB-

-RRB-

Hierarchical

classification

and

vector

quantization

with

neural

trees

Neurocomputing

-LCB-

Merz

C.

J.

and

Murphy

P.

M.

-LRB-

-RRB-

UCI

Repository

of

Machine

Learning

Databases

-LSB-

Technical

report

Department

of

Information

and

Computer

Science

University

of

California

Irvine

CA

Miyata

Y.

-LRB-

-RRB-

PlaNet

A

Tool

for

Constructing

Running

and

Looking

into

a

PDP

Network

Moody

J.

and

Darken

C.

J.

-LRB-

-RRB-

Fast

Learning

in

Networks

of

Locally-Tuned

Processing

Units

Neural

Computation

-LCB-

Moore

B.

-LRB-

-RRB-

and

Pattern

Clustering

In

Touretzky

D.

Hinton

G.

and

Sejnowski

T.

editors

Proceedings

of

the

Connectionist

Models

Summer

School

pages

-LCB-

Morgan

Kaufmann

San

Mateo

CA

Oja

E.

-LRB-

-RRB-

Neural

networks

principal

components

and

subspaces

International

Journal

of

Neural

Systems

-LCB-

Rumelhart

D.

McClelland

J.

and

the

PDP

Research

Group

-LRB-

-RRB-

Parallel

Distributed

Processing

Explorations

in

the

Microstructure

of

Cognition

volume

Foundations

chapter

pages

-LCB-

The

MIT

Press

Shepherd

G.

M.

-LRB-

-RRB-

The

Synaptic

Organization

of

the

Brain

Oxford

University

Press

New

York

Soon

H.

and

Tan

A.

-LRB-

-RRB-

Concept

hierarchy

network

for

inheritance

systems

Concept

formation

property

inheritance

and

conflict

resolution

In

Proceedings

of

Annual

Conference

of

Cognitive

Science

Society

pages

-LCB-

Lawrence

Erlbaum

Associates

Van

Essen

D.

C.

Anderson

C.

H.

and

J.

F.

D.

-LRB-

-RRB-

Information

processing

in

the

primate

visual

system

An

integrated

systems

perspective

Science

-LRB-

-RRB-

-LCB-

Wallace

C.

and

Boulton

D.

-LRB-

-RRB-

An

information

measure

for

classification

The

Computer

Journal

-LRB-

-RRB-

Williamson

J.

R.

-LRB-

-RRB-

Gaussian

ARTMAP

A

Neural

Network

for

Fast

Incremental

Learning

of

Noisy

Multidimensional

Maps

Technical

Report

CAS/CNS

Boston

University

Center

of

Adaptive

Systems

and

Department

of

Cognitive

and

Neural

Systems

adaptive resonance theory
hierarchical art
hierarchical clustering
modular networks