VICTORIA

UNIVERSITY

OF

WELLINGTON

Te

Whare

Wananga

o

te

Upoko

o

te

Ika

a

Maui

VUW

Department

of

Computer

Science

PO

Box

Wellington

New

Zealand

Tel

Fax

Internet

Tech.Reports@comp.vuw.ac.nz

Inferring

Hierarchical

Categories

with

ART-based

Modular

Neural

Networks

Guszti

Bartfai

Technical

Report

May

Abstract

This

paper

presents

a

generic

modular

neural

network

architecture

built

up

of

Adaptive

Resonance

Theory

-LRB-

ART

-RRB-

networks

The

network

has

a

more

powerful

knowledge

representation

than

its

component

ART

networks

in

that

it

is

capable

of

representing

class

hierarchies

as

opposed

to

simply

partitioning

the

input

data

set

At

the

same

time

the

network

retains

the

main

properties

of

ART

networks

like

fast

and

stable

incremental

learning

parallel

search

and

so

on

Three

different

implementations

of

the

generic

network

-LRB-

SMART

HART-J

and

HART-S

-RRB-

are

described

each

being

capable

of

unsupervised

learning

of

hierarchical

clusterings

of

arbitrary

sequences

of

input

patterns

Each

hierarchical

ART

network

produces

slightly

different

clusterings

due

to

the

differences

in

the

interactions

between

the

component

ART

modules

and

the

input

each

module

receives

Two

examples

produced

by

computer

simulations

illustrate

the

developed

and

clusterings

on

a

machine

learning

benchmark

dataset

The

architectures

are

also

general

enough

that

the

types

of

ART

modules

can

be

changed

-LRB-

e.g.

from

binary

ART

to

Fuzzy

ART

-RRB-

without

affecting

the

main

properties

of

the

network

to

suit

the

problem

domain

better

We

also

ask

questions

about

how

this

model

might

be

extended

and/or

modified

to

make

it

suitable

for

supervised

learning

to

make

better

use

of

the

structured

knowledge

present

in

the

network

to

make

the

resulting

clusterings

more

robust

and

to

create

new

ART

modules

during

learning

We

speculate

that

all

these

could

be

achieved

by

implementing

appropriate

control

mechanism

in

the

networks

Publishing

Information

This

paper

has

been

accepted

for

presentation

at

the

Neural

Networks

and

Structured

Knowledge

-LRB-

NNSK

-RRB-

workshop

of

the

European

Conference

on

Artificial

Intelligence

-LRB-

ECAI

-RRB-

in

Budapest

Hungary

Author

Information

Guszti

Bartfai

is

a

lecturer

at

the

Department

of

Computer

Science

He

welcomes

any

comments

and

suggestions

about

this

paper

His

e-mail

address

is

guszti@comp.vuw.ac.nz

Introduction

Since

the

re-emergence

of

neural

computing

in

the

late

's

much

progress

have

been

made

towards

closing

the

gap

between

symbolic

AI

and

neural

networks

Cross-fertilisation

of

ideas

has

been

fruitful

Hybrid

systems

that

include

both

AI

and

neural

parts

have

also

shown

to

be

capable

to

achieve

more

than

either

could

on

its

own

-LRB-

see

for

example

-LSB-

-RSB-

-RRB-

In

order

to

advance

further

we

need

to

identify

areas

in

which

either

side

is

weaker

than

the

other

Such

a

weak

area

for

neural

networks

is

the

treatment

of

structured

knowledge

Most

of

the

classical

networks

like

the

back-propagation

network

do

not

normally

store

structure

explicitly

and

typically

learn

to

associate

input

patterns

with

desired

target

categories

based

on

statistical

properties

The

only

evident

structure

they

have

is

the

connection

topology

and

whatever

can

be

represented

with

a

set

of

connection

weights

and

-LRB-

often

distributed

-RRB-

patterns

of

activity

The

discovered

underlying

knowledge

is

often

buried

in

the

hidden

layer

-LRB-

s

-RRB-

However

the

brain

exhibits

more

structuredness

with

certain

areas

playing

specific

roles

in

information

processing

For

example

neurons

of

the

neocortex

where

most

high-level

cognitive

processes

occur

are

of

few

types

and

their

connections

are

organised

in

relatively

few

schemes

or

basic

circuits

-LSB-

-RSB-

The

primate

visual

system

has

also

been

found

to

be

highly

modularised

with

extensively

interconnected

subdivisions

performing

linear

and

nonlinear

filtering

object

recognition

and

so

on

-LSB-

-RSB-

There

are

various

kinds

of

structured

knowledge

that

can

not

be

represented

straightforwardly

with

BP-like

networks

or

the

kinds

of

representation

they

discover

is

not

particularly

suitable

for

describing

the

problem

domain

and/or

interpreting

by

humans

Such

more

complex

structures

are

higher-order

relationships/dependencies

among

input

attributes

spatial/geometrical

relationships

class

hierarchies

and

so

on

In

this

paper

we

concentrate

on

one

type

of

structured

knowledge

representing

and

learning

of

hierarchical

categories

In

many

applications

extracting

class

hierarchies

provides

distinctive

advantages

like

easier

human

interpretation

database

design/query

Hierarchical

clustering

has

also

been

found

in

vivo

An

example

for

such

a

network

is

discussed

in

-LSB-

-RSB-

where

it

was

shown

that

the

paleocortex

performs

hierarchical

clustering

The

authors

also

hypothesise

that

networks

and

circuitry

of

similar

design

in

other

brain

regions

contain

computational

elements

sufficient

to

construct

perceptual

hierarchies

for

use

in

recognizing

environmental

cues

and

suggest

the

possibility

that

this

biologically

generated

mechanism

for

hierarchical

clustering

may

be

a

routine

part

of

perceptual

recognition

memory

behavior

in

animals

and

humans

It

is

well-known

that

a

competitive

layer

of

neurons

is

capable

of

performing

unsupervised

classification

of

input

patterns

There

have

been

individual

separate

attempts

to

break

out

and

use

some

sort

of

hierarchical

knowledge

to

achieve

certain

specific

goals

-LRB-

see

for

example

-LSB-

-RSB-

-RRB-

Besides

the

lack

of

generality

however

they

also

have

several

drawbacks

like

long

training

time

finding

and

getting

stuck

in

suboptimal

solutions

and

instability

against

changes

in

the

environment

An

alternative

model

that

is

based

on

solid

neurophysiological

and

cognitive

principles

and

also

overcomes

the

above

problems

is

the

Adaptive

Resonance

Theory

-LRB-

ART

-RRB-

neural

network

-LSB-

-RSB-

They

are

more

suitable

for

the

systematic

construction

of

modular

networks

because

of

their

well-defined

interfaces

as

well

as

features

that

most

other

networks

lack

In

specific

ART

networks

have

the

ability

to

create

new

output

nodes

-LRB-

i.e.

categories

-RRB-

dynamically

and

do

not

suffer

from

the

problem

of

forgetting

previously

learned

categories

if

the

environment

changes

In

other

words

it

provides

a

solution

to

the

stability-plasticity

dilemma

-LSB-

-RSB-

Despite

their

useful

properties

however

ART

networks

by

themselves

can

only

learn

input

categories

at

a

given

level

of

specificity

-LRB-

i.e.

partition

the

input

space

-RRB-

which

de

Inputcategory

layercomparison

layerinput

layer

Attentional

subsystemOrienting

subsystem

Figure

Architecture

of

the

ART

network

Thick

arrows

denote

combined

or

non-specific

signals

Winning

category

nodes

are

selected

by

the

attentional

subsystem

Category

search

is

controlled

by

the

orienting

subsystem

If

the

degree

of

category

match

at

the

layer

is

lower

than

the

vigilance

level

ae

the

non-specific

reset

signal

will

be

triggered

which

will

deactivate

the

current

winning

node

for

the

period

of

presentation

of

the

current

input

pends

on

a

global

parameter

called

vigilance

In

this

paper

we

discuss

networks

built

up

of

ART

modules

that

are

capable

of

inferring

class

hierarchies

from

input

data

The

representational

capabilities

of

each

hierarchical

model

are

demonstrated

by

computer

simulations

First

we

review

the

ART

network

in

section

The

hierarchical

ART

models

and

their

properties

are

discussed

in

section

Conclusions

are

drawn

in

section

after

some

discussion

in

section

in

which

several

questions

are

raised

relating

to

the

representation

learning

mechanism

and

applicability

of

hierarchical

ART

networks

ART

Neural

Networks

This

section

describes

the

ART

architecture

at

a

level

of

detail

that

is

necessary

for

understanding

the

rest

of

the

paper

Unless

otherwise

stated

we

refer

to

the

binary

version

of

ART

-LSB-

-RSB-

An

ART

network

is

built

up

of

three

layers

the

input

layer

-LRB-

-RRB-

the

comparison

layer

-LRB-

-RRB-

and

the

recognition

layer

-LRB-

-RRB-

with

N

N

and

M

neurons

respectively

-LRB-

see

figure

-RRB-

The

input

layer

stores

the

input

pattern

and

each

neuron

in

the

input

layer

is

connected

to

its

corresponding

node

in

the

comparison

layer

via

one-to-one

non-modifiable

links

Nodes

in

the

layer

represent

input

categories

The

and

layers

interact

with

each

other

through

weighted

bottom-up

and

top-down

connections

that

are

modified

when

the

network

learns

There

are

additional

gain

control

signals

in

the

network

-LRB-

not

shown

in

figure

-RRB-

that

regulate

its

operation

but

those

will

not

be

detailed

here

The

learning

process

of

the

network

can

be

described

as

follows

At

each

presentation

of

a

non-zero

binary

input

pattern

x

-LRB-

xj

j

N

-RRB-

are

ART

networks

that

accept

both

continuous

and

binary

inputs

-LSB-

-RSB-

the

network

attempts

to

classify

it

into

one

of

its

existing

categories

based

on

its

similarity

to

the

stored

prototype

of

each

category

node

More

precisely

for

each

node

i

in

the

layer

the

bottom-up

activation

Ti

is

calculated

which

can

be

expressed

as

Ti

jwi

xj

jwij

i

M

-LRB-

-RRB-

where

j

j

is

the

norm

operator

-LRB-

jxj

PNj

xj

-RRB-

wi

is

the

-LRB-

binary

-RRB-

weight

vector

-LRB-

or

prototype

-RRB-

of

category

i

and

is

the

choice

parameter

-LSB-

-RSB-

Then

the

node

I

that

has

the

highest

bottom-up

activation

i.e.

TI

maxfTiji

Mg

is

selected

-LRB-

winnertake-all

competition

-RRB-

The

weight

vector

of

the

winning

node

-LRB-

wI

-RRB-

will

then

be

compared

to

the

current

input

at

the

comparison

layer

If

they

are

similar

enough

i.e.

they

satisfy

the

jwI

xj

jxj

ae

-LRB-

-RRB-

matching

condition

where

ae

is

a

system

parameter

called

vigilance

-LRB-

ae

-RRB-

node

I

will

capture

the

current

input

and

the

network

learns

by

modifying

wI

wnew

I

-LRB-

wold

I

x

-RRB-

-LRB-

-RRB-

wold

I

-LRB-

-RRB-

where

is

the

learning

rate

-LRB-

-RRB-

All

other

weights

in

the

network

remain

unchanged

If

however

the

stored

prototype

wI

does

not

match

the

input

sufficiently

i.e.

condition

-LRB-

-RRB-

is

not

met

the

winning

node

will

be

reset

-LRB-

by

activating

the

reset

signal

in

figure

-RRB-

for

the

period

of

presentation

of

the

current

input

Then

another

node

-LRB-

or

category

-RRB-

is

selected

with

the

highest

Ti

whose

prototype

will

be

matched

against

the

input

and

so

on

This

hypothesis-testing

cycle

is

repeated

until

the

network

either

finds

a

stored

category

whose

prototype

matches

the

input

well

enough

or

allocates

a

new

node

Then

learning

takes

place

according

to

-LRB-

-RRB-

It

can

be

shown

that

after

an

initial

period

of

self-stabilisation

the

network

will

directly

-LRB-

i.e.

without

search

-RRB-

access

the

prototype

of

one

of

the

categories

it

has

found

in

a

given

training

set

The

number

of

developed

categories

can

be

controlled

by

setting

ae

the

higher

the

vigilance

level

the

larger

number

of

more

specific

categories

will

be

created

-LRB-

If

ae

the

network

will

create

a

new

category

for

every

unique

input

-RRB-

Also

as

a

consequence

of

its

stability-plasticity

property

the

network

is

capable

of

learning

on-line

i.e.

refining

its

learned

categories

in

response

to

a

stream

of

input

patterns

-LRB-

as

opposed

to

being

trained

off-line

on

a

finite

training

set

-RRB-

With

an

ART

network

therefore

we

can

find

classes

in

an

input

set

at

a

given

level

of

specificity

If

we

want

a

more

general

view

of

the

input

set

we

need

to

train

an

ART

network

with

low

vigilance

for

a

more

detailed

view

we

need

to

retrain

the

network

with

high

vigilance

or

simply

increase

vigilance

during

training

In

any

case

however

the

network

will

only

be

able

to

see

an

input

set

at

a

given

level

of

specificity

at

any

given

time

There

is

no

relationship

between

any

pairs

of

category

prototypes

except

that

they

are

in

the

same

network

and

compete

with

each

other

With

only

one

level

of

competitive

layer

the

network

is

clearly

not

capable

of

representing

structured

knowledge

simplicity

we

assume

that

the

network

is

in

fast

learning

mode

-LRB-

see

below

-RRB-

in

which

case

the

bottom-up

and

top-down

weights

for

each

category

i

are

identical

-LRB-

wi

see

-LSB-

-RSB-

-RRB-

Figure

Architecture

of

a

two-layer

smart

network

It

is

composed

of

two

ART

modules

-LRB-

in

layer

and

layer

-RRB-

whose

comparison

layers

-LRB-

and

-RRB-

are

connected

via

associative

links

Both

modules

perform

clustering

of

the

same

input

patterns

and

learn

to

associate

categories

developed

at

different

vigilance

levels

The

module

attempts

to

predict

the

category

at

If

it

fails

the

module

will

raise

the

vigilance

level

at

-LRB-

through

an

internal

feedback

mechanism

-RRB-

to

cause

a

mismatch

at

layer

which

will

subsequently

search

for

a

better

i.e.

more

consistent

category

Hierarchical

ART

Networks

In

this

section

we

discuss

three

modular

ART-based

networks

we

have

proposed

which

are

capable

of

inferring

class

hierarchies

from

a

training

set

These

are

demonstrated

by

results

of

computer

simulations

where

the

networks

were

trained

on

the

Zoo

machine

learning

benchmark

database

-LSB-

-RSB-

The

first

model

is

the

concept

of

Self-consistent

Modular

ART

-LRB-

smart

-RRB-

network

-LSB-

-RSB-

The

architecture

of

a

two-layer

smart

network

is

shown

in

figure

In

this

network

two-level

class

hierarchies

are

learnt

by

making

the

and

modules

develop

broader

and

more

specific

classes

respectively

-LRB-

by

setting

-RRB-

Each

category

will

have

one

or

more

sub-categories

but

no

two

categories

share

the

same

category

This

self-consistent

hierarchy

is

represented

explicitly

in

the

network

through

the

associative

links

Table

shows

a

typical

internal

representation

of

the

smart

network

-LRB-

The

actual

clusterings

varied

depending

on

the

order

of

presentation

of

the

input

patterns

in

each

training

session

-RRB-

Columns

-LRB-

j

-RRB-

show

the

category

prototypes

and

columns

-LRB-

j

-RRB-

contain

the

prototypes

of

categories

In

this

example

three

sub-classes

were

developed

at

layer

with

nodes

and

belonging

to

parent

classes

and

respectively

in

layer

The

table

shows

that

there

is

a

clear

hierarchy

in

the

network

in

that

a

more

general

class

is

connected

to

several

sub-classes

that

inherit

some

features

from

their

parent

class

as

well

as

contain

features

specific

to

that

particular

sub-class

We

can

also

see

that

a

feature

has

a

specific

value

in

an

prototype

if

and

only

if

all

its

sub-classes

have

that

feature

set

to

the

same

value

-LRB-

e.g.

features

breathes

and

tail

in

-RRB-

The

next

architecture

is

hart-j

a

hierarchical

ART

network

that

is

capable

of

learning

class

hierarchies

by

joining

-LRB-

more

specific

-RRB-

sub-classes

to

form

more

general

classes

-LSB-

-RSB-

The

architecture

of

a

two-layer

hart-j

network

is

shown

in

figure

Both

layers

in

a

hart-j

network

perform

clustering

of

their

respective

input

patterns

However

unlike

the

smart

can

be

generalised

to

more

than

two

layers

but

this

will

not

be

discussed

here

Category

prototype

vectors

Attributes

hair

-LCB-

-LCB-

-LCB-

-LCB-

no

no

-LCB-

no

-LCB-

no

no

-LCB-

-LCB-

-LCB-

no

no

feathers

-LCB-

-LCB-

no

-LCB-

yes

no

no

no

no

no

no

no

no

no

no

no

eggs

-LCB-

-LCB-

no

-LCB-

yes

yes

-LCB-

yes

yes

-LCB-

yes

-LCB-

no

yes

yes

yes

milk

-LCB-

-LCB-

-LCB-

-LCB-

no

no

no

no

no

no

no

-LCB-

yes

-LCB-

no

no

airborne

-LCB-

-LCB-

no

-LCB-

yes

no

-LCB-

no

-LCB-

no

no

no

no

no

no

no

aquatic

-LCB-

-LCB-

no

-LCB-

no

no

-LCB-

yes

no

-LCB-

no

-LCB-

-LCB-

yes

yes

-LCB-

predator

-LCB-

-LCB-

-LCB-

-LCB-

-LCB-

yes

-LCB-

-LCB-

-LCB-

yes

yes

-LCB-

-LCB-

-LCB-

-LCB-

-LCB-

toothed

-LCB-

-LCB-

-LCB-

-LCB-

no

yes

-LCB-

-LCB-

no

yes

no

-LCB-

yes

-LCB-

yes

-LCB-

backbone

-LCB-

yes

-LCB-

yes

yes

yes

-LCB-

-LCB-

-LCB-

yes

no

yes

yes

yes

yes

yes

breathes

yes

yes

yes

yes

yes

yes

-LCB-

-LCB-

yes

-LCB-

no

-LCB-

yes

-LCB-

no

yes

venomous

-LCB-

no

-LCB-

no

no

yes

-LCB-

-LCB-

-LCB-

yes

no

no

no

no

no

no

fins

-LCB-

no

no

-LCB-

no

no

-LCB-

-LCB-

no

no

no

-LCB-

-LCB-

-LCB-

yes

no

legs

-LCB-

-LCB-

-LCB-

-LCB-

-LCB-

-LCB-

-LCB-

-LCB-

-LCB-

-LCB-

tail

yes

yes

yes

yes

yes

yes

-LCB-

-LCB-

-LCB-

yes

no

-LCB-

-LCB-

yes

yes

-LCB-

domestic

-LCB-

no

-LCB-

no

-LCB-

no

-LCB-

no

-LCB-

no

no

-LCB-

-LCB-

-LCB-

no

no

catsize

-LCB-

no

-LCB-

yes

no

no

-LCB-

-LCB-

-LCB-

no

no

-LCB-

-LCB-

-LCB-

-LCB-

-LCB-

Category

size

Table

A

typical

example

for

the

developed

internal

representation

in

the

smart

network

Instances

-LRB-

of

animals

-RRB-

are

described

with

boolean

and

numeric

attributes

A

-LCB-

attribute

value

should

be

interpreted

as

do

n't

care

Layer

and

layer

vigilance

levels

were

and

respectively

In

this

training

run

the

network

developed

and

categories

at

layers

and

respectively

The

number

of

input

patterns

in

each

category

is

shown

in

the

bottom

row

InputLayer

Figure

Architecture

of

a

two-layer

hart-j

network

All

vertical

connections

are

one-to-one

The

input

layer

of

is

identical

to

the

comparison

layer

of

breathesno

finshas

taillays

eggs

no

milkno

feathersnot

airborne

toothedhas

backbone

not

domesticno

hairhas

backbone

not

venomous

has

tailnot

catsize

no

feathers

not

toothed

no

backbone

no

finsno

tail

no

eggspredatorno

finshas

tailbreathesno

finsno

tailpredatornot

venomous

not

domestic

not

aquatic

not

predator

has

backbone

not

venomous

no

hairno

milknot

toothed

not

domestic

lays

eggs

not

airborne

predatorhas

backbone

has

tail

Layer

Figure

An

example

of

a

class

hierarchy

found

by

a

three-layer

HART

network

Each

box

represents

a

category

and

contains

a

list

of

critical

and

distinctive

features

of

that

class

All

other

attributes

not

shown

were

do

n't

care

For

example

class

in

layer

has

feathers

and

domestic

as

critical

features

-LRB-

both

are

absent

-RRB-

with

domestic

being

the

distinctive

one

-LRB-

from

class

-RRB-

at

that

level

Vigilance

levels

at

layers

and

were

and

respectively

network

the

module

learns

to

cluster

not

the

input

patterns

but

the

category

prototypes

the

module

develops

in

response

to

the

original

input

patterns

This

way

the

module

develops

broader

or

more

general

categories

than

if

-LSB-

-RSB-

The

hierarchy

is

not

represented

explicitly

-LRB-

like

in

the

smart

network

-RRB-

but

emerges

as

input

patterns

are

presented

Expanding

the

hart-j

network

to

more

layers

is

relatively

straightforward

-LRB-

There

is

also

a

limit

to

the

number

of

layers

needed

on

a

given

training

set

-LSB-

-RSB-

-RRB-

As

an

example

figure

depicts

the

class

hierarchy

that

a

three-layer

hart-j

network

found

during

a

training

session

It

shows

how

categories

become

increasingly

specific

-LRB-

with

more

critical

features

-RRB-

at

lower

levels

Note

the

empty

boxes

-LRB-

and

-RRB-

in

layer

which

means

that

those

categories

were

found

identical

to

their

parents

-LRB-

and

in

layer

-RRB-

at

the

given

vigilance

level

in

layer

Only

at

layer

-LRB-

with

higher

vigilance

-RRB-

were

those

categories

split

up

into

smaller

more

specific

ones

-LRB-

We

observed

that

not

all

developed

clusterings

looked

realistic

This

only

indicates

that

using

the

given

attributes

and

without

a

bias

or

weightings

of

any

of

the

features

-LSB-

which

humans

certainly

have

-RSB-

those

were

plausible

classes

for

the

network

to

discover

-RRB-

The

third

architecture

-LRB-

hart-s

-LSB-

-RSB-

-RRB-

is

a

modular

ART

network

whose

overall

architecture

is

similar

to

hart-j

However

here

the

idea

is

to

form

the

most

general

level

of

clusters

first

and

then

successively

splitting

those

clusters

into

more

specific

ones

Only

the

difference

between

an

input

to

a

particular

ART

module

and

the

prototype

of

the

category

of

that

input

is

propagated

to

the

next

ART

module

The

input

signal

to

the

first

module

is

the

difference

between

the

input

pattern

and

the

most

general

category

-LRB-

which

contains

all

training

examples

-RRB-

so

is

equivalent

to

the

input

pattern

itself

Only

when

such

differences

are

greater

than

a

size

determined

by

a

network

parameter

called

resolution

is

the

next

ART

module

activated

This

new

parameter

allows

us

to

control

the

depth

of

a

classification

tree

while

we

can

think

of

vigilance

levels

as

parameters

that

affect

the

width

of

the

tree

A

distinctive

feature

of

the

hart-s

network

is

the

existence

adaptive

associative

connections

between

the

layers

of

successive

ART

modules

which

provides

the

links

between

any

cluster

and

its

immediate

sub-clusters

When

a

category

is

active

in

the

first

of

a

pair

of

modules

only

those

categories

in

the

second

module

that

can

be

sub-categories

of

the

first

are

allowed

to

become

active

These

links

are

important

in

this

network

-LRB-

which

are

not

needed

in

hart-j

-RRB-

so

the

correct

hierarchies

can

be

maintained

since

the

individual

ART

modules

cluster

differences

only

Category

prototypes

are

often

duplicated

in

hart-j

as

prototypes

are

propagated

on

whether

or

not

any

clustering

of

them

is

achieved

Prototypes

in

hart-s

by

comparison

always

differ

and

furthermore

information

contained

in

one

prototype

is

not

copied

in

subcluster

prototypes

as

occurs

hart-j

A

difficulty

with

hart-s

is

in

predicting

the

number

of

nodes

needed

in

lower

layers

of

a

network

due

to

the

links

created

in

hierarchical

memory

during

learning

A

higher

setting

of

the

resolution

parameter

tends

to

reduce

the

number

of

such

nodes

used

to

learn

a

particular

set

of

inputs

Also

worth

noting

is

the

difference

in

the

way

the

component

ART

networks

interact

in

the

smart

and

hart

networks

In

the

smart

network

each

ART

module

receives

the

original

input

pattern

and

processes

it

in

parallel

As

a

consequence

the

hierarchy

of

developed

categories

must

be

assured

explicitly

otherwise

a

class

at

one

level

can

belong

to

two

or

more

classes

at

a

higher

level

This

consistency

is

maintained

via

associative

links

between

categories

at

subsequent

levels

and

an

internal

feedback

mechanism

This

however

results

in

the

smart

network

developing

many

more

categories

at

lower

levels

than

comparable

hart

networks

-LSB-

-RSB-

This

drawback

might

be

expected

to

be

offset

by

the

fact

that

the

component

ART

modules

work

in

parallel

as

opposed

to

hart

where

each

ART

module

must

wait

until

the

previous

layer

has

finished

searching

for

a

suitable

category

for

its

input

It

turns

out

to

be

not

true

however

because

the

smart

network

must

spend

a

comparable

amount

of

time

during

the

initial

period

of

training

to

resolve

conflicts

due

to

mismatching

-LRB-

nonhierarchical

-RRB-

categories

The

total

training

time

-LRB-

in

average

number

of

epochs

-RRB-

was

found

to

be

very

similar

-LSB-

-RSB-

In

conclusion

it

appears

that

the

hart-j

and

hart-s

networks

offer

an

overall

better

architecture

for

learning

hierarchical

clusterings

Discussion

A

discussion

on

the

hart-j

and

hart-s

networks

their

properties

and

their

relationship

to

other

neural

network

models

is

presented

in

-LSB-

-RSB-

One

of

the

main

distinctive

features

of

the

hierarchical

ART

networks

we

have

proposed

is

that

they

were

designed

with

the

explicit

goal

of

going

beyond

simple

partitioning

of

the

input

space

in

mind

We

see

them

as

an

attempt

to

create

networks

that

have

an

increased

representational

power

while

retaining

the

common

properties

of

artificial

neural

networks

-LRB-

e.g.

learning

generalisation

parallel

distributed

processing

etc.

-RRB-

These

modular

ART

networks

have

a

more

powerful

knowledge

representation

than

single

competitive

learning

networks

-LRB-

hierarchical

clustering

vs.

partitioning

-RRB-

the

same

way

multi-layer

Perceptrons

are

more

powerful

than

single-layer

ones

-LRB-

e.g.

representing

arbitrary

boolean

functions

as

opposed

to

linearly

separable

ones

-RRB-

Furthermore

the

network

architecture

is

not

application

specific

-LRB-

unlike

NIRS

-LSB-

-RSB-

for

example

-RRB-

which

means

that

it

will

probably

work

on

many

problem

domains

where

the

task

is

to

learn

stable

clusterings

of

-LRB-

potentially

infinite

-RRB-

sequences

of

input

data

A

real-world

application

of

a

network

similar

to

hart-j

is

described

in

-LSB-

-RSB-

We

are

currently

working

on

testing

the

hierarchical

ART

networks

on

larger

databases

and

are

also

looking

at

the

comparison

of

hierarchical

ART

networks

against

statistical

and

machine

learning

clustering

algorithms

There

are

a

number

of

issues

and

questions

that

have

been

raised

as

a

result

of

our

investigations

ffl

How

can

these

networks

be

used

in

supervised

learning

of

target

categories

-LRB-

or

arbitrary

output

patterns

-RRB-

We

think

that

this

is

possible

by

injecting

an

appropriate

control

mechanisms

-LRB-

via

for

example

non-specific

control

signals

like

the

ones

in

ART

-RRB-

into

the

architecture

to

guide

its

search

and

prediction

as

well

as

to

correct

itself

after

making

a

wrong

prediction

ffl

How

can

clusterings

be

made

more

robust

i.e.

less

dependent

on

the

order

of

input

presentation

Although

we

are

not

convinced

that

this

is

always

a

desirable

goal

we

think

that

the

architecture

is

generic

enough

that

this

could

be

achieved

by

plugging

in

more

suitable

network

models

-LRB-

e.g.

Gaussian

ARTMAP

network

-LSB-

-RSB-

-RRB-

ffl

In

what

other

ways

can

structured

knowledge

represented

in

the

network

be

used

We

believe

this

richer

representation

allows

the

network

for

example

to

make

use

of

its

categories

at

multiple

levels

to

enhance

its

prediction

performance

It

could

also

implement

a

more

complex

learning

and

search

-LRB-

matching

-RRB-

mechanism

that

involve

multiple

layers

-LRB-

e.g.

a

higher

level

guiding

the

search

of

a

lower

one

by

providing

feedback

to

it

-RRB-

ffl

How

can

the

dynamic

creation

-LRB-

and

freeing

up

-RRB-

of

whole

ART

modules

be

achieved

-LRB-

thus

the

network

being

able

to

change

the

granularity

of

clustering

during

learning

-RRB-

We

believe

this

would

be

a

very

useful

property

but

also

think

that

this

may

not

be

achieved

unless

the

network

has

an

explicit

utility

function

that

it

maximises

and

therefore

drives

its

learning

process

much

like

machine

learning

mechanisms

do

-LRB-

e.g.

COBWEB

-LSB-

-RSB-

-RRB-

One

way

to

implement

dynamic

creation

of

new

levels

is

to

treat

whole

ART

modules

the

same

way

as

individual

nodes

are

treated

within

a

single

ART

network

-LRB-

by

letting

a

pool

of

initially

uncommitted

nodes

take

part

in

the

competition

-RRB-

thereby

creating

a

self-similar

network

architecture

-LRB-

which

is

one

of

the

advantages

of

structuredness

-LSB-

-RSB-

-RRB-

Conclusion

In

this

paper

we

discussed

three

hierarchical

network

architecture

built

up

of

ART

modules

that

are

capable

of

learning

stable

hierarchical

clusterings

of

arbitrary

sequences

of

input

patterns

The

internal

representations

of

these

networks

demonstrated

-LRB-

-RRB-

hierarchies

of

classes

where

lower-level

classes

inherited

features

from

ones

at

higher-levels

and

had

their

own

distinctive

features

as

well

We

believe

that

the

ideas

and

results

presented

here

show

the

benefits

of

modularity

in

learning

more

complex

relationships

from

the

input

environment

We

hope

that

the

issues

raised

here

will

stimulate

discussions

-LRB-

which

may

raise

further

questions

-RRB-

among

participants

and

so

this

paper

will

contribute

to

the

success

of

this

workshop

Acknowledgements

I

would

like

to

thank

the

referees

for

their

comments

which

helped

improve

this

paper

References

-LSB-

-RSB-

J.

Ambros-Ingerson

R.

Granger

and

G.

Lynch

Simulation

of

paleocortex

performs

hierarchical

clustering

Science

-LCB-

-LSB-

-RSB-

G.

Bartfai

Hierarchical

Clustering

with

ART

Neural

Networks

In

Proceedings

of

the

IEEE

International

Conference

on

Neural

Networks

volume

pages

-LCB-

IEEE

Press

-LSB-

-RSB-

G.

Bartfai

A

Comparison

of

Two

ART-based

Neural

Networks

for

Hierarchical

Clustering

In

K.

Kasabov

and

G.

Coghill

editors

ANNES

The

Second

New

Zealand

International

Two-Stream

Conference

on

Artificial

Neural

Networks

and

Expert

Systems

pages

-LCB-

IEEE

Computer

Society

Press

November

-LSB-

-RSB-

G.

Bartfai

An

ART-based

Modular

Architecture

for

Learning

Hierarchical

Clusterings

Neurocomputing

In

press

-LSB-

-RSB-

G.

Bartfai

and

R.

White

ART-based

Modular

Networks

for

Incremental

Learning

of

Hierarchical

Clusterings

Connection

Science

Submitted

-LSB-

-RSB-

G.A.

Carpenter

and

S.

Grossberg

A

Massively

Parallel

Architecture

for

a

SelfOrganizing

Neural

Pattern

Recognition

Machine

Computer

Vision

Graphics

and

Image

Processing

-LCB-

-LSB-

-RSB-

G.A.

Carpenter

and

S.

Grossberg

Self-Organizing

of

Stable

Category

Recognition

Codes

for

Analog

Input

Patterns

Applied

Optics

-LRB-

-RRB-

-LCB-

December

-LSB-

-RSB-

G.A.

Carpenter

S.

Grossberg

and

J.H.

Reynolds

ARTMAP

Supervised

Real-Time

Learning

and

Classification

of

Nonstationary

Data

by

a

Self-Organizing

Neural

Network

Neural

Networks

-LCB-

-LSB-

-RSB-

G.A.

Carpenter

S.

Grossberg

and

D.B.

Rosen

Fuzzy

ART

Fast

Stable

Learning

and

Categorization

of

Analog

Patterns

by

an

Adaptive

Resonance

System

Neural

Networks

-LCB-

-LSB-

-RSB-

Gail

A.

Carpenter

and

Stephen

Grossberg

The

ART

of

Adaptive

Pattern

Recognition

by

a

Self-Organizing

Neural

Network

IEEE

Computer

-LRB-

-RRB-

-LCB-

March

-LSB-

-RSB-

Thomas

P.

Caudell

Scott

D.

G.

Smith

Richard

Escobedo

and

Michael

Anderson

NIRS

Large

Scale

Neural

Architectures

for

Engineering

Design

Retrieval

Neural

Networks

-LRB-

-RRB-

-LSB-

-RSB-

D.H.

Fisher

Knowledge

Acquisition

Via

Incremental

Conceptual

Clustering

Machine

Learning

-LRB-

-RRB-

-LCB-

September

-LSB-

-RSB-

B.

Happel

and

J.

Murre

Design

and

evolution

of

modular

neural

network

architectures

Neural

Networks

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

Shigekazu

Ishihara

Keiko

Ishihara

Mistsuo

Nagamachi

and

Yukihiro

Matsubara

arboART

ART

based

hierarchical

clustering

and

its

application

to

questionnaire

data

analysis

In

Proceedings

of

the

IEEE

International

Conference

on

Neural

Networks

volume

pages

-LCB-

IEEE

Press

-LSB-

-RSB-

Jouko

Lampinen

and

Erkki

Oja

Clustering

properties

of

hierarchical

self-organizing

maps

Journal

of

Mathematical

Imaging

and

Vision

-LCB-

-LSB-

-RSB-

Tao

Li

Luyuan

Fang

and

Ken

Q-Q

Li

Hierarchical

classification

and

vector

quantization

with

neural

trees

Neurocomputing

-LCB-

-LSB-

-RSB-

C.

J.

Merz

and

P.

M.

Murphy

UCI

Repository

of

Machine

Learning

Databases

-LSB-

Technical

report

Department

of

Information

and

Computer

Science

University

of

California

Irvine

CA

-LSB-

-RSB-

Dean

A.

Pomerleau

Neural

network

perception

for

mobile

robot

guidance

Kluwer

Academic

Publishers

-LSB-

-RSB-

D.E.

Rumelhart

J.L.

McClelland

and

the

PDP

Research

Group

Parallel

Distributed

Processing

Explorations

in

the

Microstructure

of

Cognition

volume

Foundations

chapter

pages

-LCB-

The

MIT

Press

-LSB-

-RSB-

G.

M.

Shepherd

The

Synaptic

Organization

of

the

Brain

Oxford

University

Press

New

York

-LSB-

-RSB-

David

C.

Van

Essen

Charles

H.

Anderson

and

Felleman

Daniel

J.

Information

processing

in

the

primate

visual

system

An

integrated

systems

perspective

Science

-LRB-

-RRB-

-LCB-

January

-LSB-

-RSB-

J.

R.

Williamson

Gaussian

ARTMAP

A

Neural

Network

for

Fast

Incremental

Learning

of

Noisy

Multidimensional

Maps

Technical

Report

CAS/CNS

Boston

University

Center

of

Adaptive

Systems

and

Department

of

Cognitive

and

Neural

Systems

February

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O B O O O O O O O O O B I O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O B I O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O B I O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

adaptive resonance theory
art
hierarchical clustering
knowledge representation