Report

NEURAL

NETWORKS

WITH

REAL

WEIGHTS

ANALOG

COMPUTATIONAL

COMPLEXITY

Hava

T.

Siegelmann

Department

of

Computer

Science

Rutgers

University

New

Brunswick

NJ

E-mail

siegelma@yoko.rutgers.edu

Eduardo

D.

Sontag

Department

of

Mathematics

Rutgers

University

New

Brunswick

NJ

E-mail

sontag@control.rutgers.edu

ABSTRACT

We

pursue

a

particular

approach

to

analog

computation

based

on

dynamical

systems

of

the

type

used

in

neural

networks

research

Our

systems

have

a

fixed

structure

invariant

in

time

corresponding

to

an

unchanging

number

of

neurons

If

allowed

exponential

time

for

computation

they

turn

out

to

have

unbounded

power

However

under

polynomial-time

constraints

there

are

limits

on

their

capabilities

though

being

more

powerful

than

Turing

Machines

-LRB-

A

similar

but

more

restricted

model

was

shown

to

be

polynomial-time

equivalent

to

classical

digital

computation

in

the

previous

work

-LSB-

-RSB-

-RRB-

Moreover

there

is

a

precise

correspondence

between

nets

and

standard

non-uniform

circuits

with

equivalent

resources

and

as

a

consequence

one

has

lower

bound

constraints

on

what

they

can

compute

This

relationship

is

perhaps

surprising

since

our

analog

devices

do

not

change

in

any

manner

with

input

size

We

note

that

these

networks

are

not

likely

to

solve

polynomially

NP-hard

problems

as

the

equality

p

np

in

our

model

implies

the

almost

complete

collapse

of

the

standard

polynomial

hierarchy

In

contrast

to

classical

computational

models

the

models

studied

here

exhibit

at

least

some

robustness

with

respect

to

noise

and

implementation

errors

Key

words

analog

computation

neural

networks

dynamical

systems

circuit

complexity

Research

supported

in

part

by

US

Air

Force

Grant

Rutgers

Center

for

Systems

and

Control

September

NEURAL

NETWORKS

WITH

REAL

WEIGHTS

ANALOG

COMPUTATIONAL

COMPLEXITY

Hava

T.

Siegelmann

Department

of

Computer

Science

Rutgers

University

New

Brunswick

NJ

E-mail

siegelma@yoko.rutgers.edu

Eduardo

D.

Sontag

Department

of

Mathematics

Rutgers

University

New

Brunswick

NJ

E-mail

sontag@control.rutgers.edu

ABSTRACT

We

pursue

a

particular

approach

to

analog

computation

based

on

dynamical

systems

of

the

type

used

in

neural

networks

research

Our

systems

have

a

fixed

structure

invariant

in

time

corresponding

to

an

unchanging

number

of

neurons

If

allowed

exponential

time

for

computation

they

turn

out

to

have

unbounded

power

However

under

polynomial-time

constraints

there

are

limits

on

their

capabilities

though

being

more

powerful

than

Turing

Machines

-LRB-

A

similar

but

more

restricted

model

was

shown

to

be

polynomial-time

equivalent

to

classical

digital

computation

in

the

previous

work

-LSB-

-RSB-

-RRB-

Moreover

there

is

a

precise

correspondence

between

nets

and

standard

non-uniform

circuits

with

equivalent

resources

and

as

a

consequence

one

has

lower

bound

constraints

on

what

they

can

compute

This

relationship

is

perhaps

surprising

since

our

analog

devices

do

not

change

in

any

manner

with

input

size

We

note

that

these

networks

are

not

likely

to

solve

polynomially

NP-hard

problems

as

the

equality

p

np

in

our

model

implies

the

almost

complete

collapse

of

the

standard

polynomial

hierarchy

In

contrast

to

classical

computational

models

the

models

studied

here

exhibit

at

least

some

robustness

with

respect

to

noise

and

implementation

errors

Introduction

Neural

networks

have

attracted

much

attention

lately

as

models

of

analog

computation

Such

nets

consist

of

a

finite

number

of

simple

processors

each

of

which

computes

a

scalar

real-valued

not

binary

function

of

an

integrated

input

This

scalar

function

or

activation

is

meant

to

reflect

the

graded

response

of

biological

neurons

to

the

net

sum

of

excitatory

and

inhibitory

inputs

affecting

them

The

existence

of

feedback

loops

in

the

interconnection

graph

gives

rise

to

a

dynamical

system

In

this

paper

we

introduce

a

mathematical

model

for

such

recurrent

neural

networks

and

we

study

their

computational

abilities

This

research

was

supported

in

part

by

US

Air

Force

Grant

Main

Results

We

focus

on

recurrent

neural

networks

In

these

networks

the

activation

of

each

processor

is

updated

according

to

a

certain

type

of

piecewise

affine

function

of

the

activations

-LRB-

xj

-RRB-

and

inputs

-LRB-

uj

-RRB-

at

the

previous

instant

with

real

coefficients

also

called

weights

-LRB-

aij

bij

ci

-RRB-

Each

processor

's

state

is

updated

by

an

equation

of

the

type

xi

-LRB-

t

-RRB-

oe

N

X

j

aijxj

-LRB-

t

-RRB-

M

X

j

bijuj

-LRB-

t

-RRB-

ci

A

i

N

-LRB-

-RRB-

where

N

is

the

number

of

processors

and

M

is

the

number

of

external

input

signals

The

function

oe

is

the

simplest

possible

sigmoid

namely

the

saturated-linear

function

oe

-LRB-

x

-RRB-

if

x

x

if

x

if

x

-LRB-

-RRB-

We

will

give

later

a

precise

definition

of

language

acceptance

for

these

computational

models

We

prove

that

neural

networks

can

recognize

in

polynomial

time

the

same

class

of

languages

as

those

recognized

by

Turing

Machines

that

consult

sparse

oracles

in

polynomial

time

-LRB-

the

class

P/poly

-RRB-

they

can

recognize

all

languages

including

of

course

non-computable

ones

in

exponential

time

Furthermore

we

show

that

almost

every

language

requires

exponential

recognition

time

-LRB-

For

simplicity

we

give

our

main

results

in

terms

of

recognition

it

is

also

possible

to

provide

a

more

general

version

regarding

the

computation

of

more

general

functions

-RRB-

The

proofs

of

the

above

results

will

be

consequences

of

the

following

equivalence

For

functions

T

IN

IN

and

S

IN

IN

let

net

-LRB-

T

-RRB-

be

the

class

of

all

functions

computed

by

neural

networks

in

time

T

-LRB-

n

-RRB-

that

is

recognition

of

strings

of

length

n

is

in

time

at

most

T

-LRB-

n

-RRB-

and

let

circuit

-LRB-

S

-RRB-

the

class

of

functions

computed

by

non-uniform

families

of

circuits

of

size

S

-LRB-

n

-RRB-

that

is

circuits

for

input

vectors

of

length

n

have

size

at

most

S

-LRB-

n

-RRB-

We

show

that

if

F

is

so

that

F

-LRB-

n

-RRB-

n

then

net

-LRB-

F

-LRB-

n

-RRB-

-RRB-

circuit

-LRB-

Poly

-LRB-

F

-LRB-

n

-RRB-

-RRB-

-RRB-

and

circuit

-LRB-

F

-LRB-

n

-RRB-

-RRB-

net

-LRB-

Poly

-LRB-

F

-LRB-

n

-RRB-

-RRB-

-RRB-

This

equivalence

will

allow

us

to

make

use

of

results

from

the

theory

of

-LRB-

nonuniform

-RRB-

circuit

complexity

We

show

that

if

one

allows

multiplications

in

addition

to

only

linear

operations

in

each

neuron

that

is

if

one

considers

instead

what

are

often

called

high

order

neural

nets

the

computational

power

does

not

increase

Even

further

and

perhaps

more

surprising

no

increase

in

computational

power

-LRB-

up

to

polynomial

time

-RRB-

can

be

achieved

by

letting

the

activation

function

be

not

necessarily

the

simple

saturated

linear

one

in

equation

-LRB-

-RRB-

but

any

function

which

satisfies

certain

reasonable

assumptions

Also

no

increase

results

even

if

the

activation

functions

are

not

necessarily

identical

in

the

different

processors

One

might

ask

about

using

such

analog

models

maybe

high

order

nets

to

solve

NP-hard

problems

in

polynomial

time

We

introduce

a

nondeterministic

model

and

show

that

the

equality

p

np

in

the

nets

model

is

very

not

likely

as

it

would

imply

the

collapse

of

the

polynomial

hierarchy

to

The

models

used

here

have

a

weak

property

of

robustness

to

noise

and

to

implementation

error

in

the

sense

that

small

enough

changes

in

the

network

would

not

affect

the

computation

The

robustness

includes

changes

in

the

precise

form

of

the

activation

function

in

the

weights

of

the

network

and

even

an

error

in

the

update

In

classical

models

of

-LRB-

digital

-RRB-

computation

this

type

of

robustness

can

not

even

be

properly

defined

A

Previous

Related

Result

In

our

previous

work

-LSB-

-RSB-

we

showed

that

if

one

restricts

to

nets

all

whose

interconnection

weights

are

rational

numbers

which

we

call

rational

nets

then

one

obtains

a

model

of

computation

that

is

polynomially

related

to

Turing

Machines

More

precisely

given

any

multi-tape

Turing

Machine

one

can

simulate

it

in

real

time

by

some

network

with

rational

weights

and

of

course

the

converse

simulation

in

polynomial

time

is

obvious

Here

we

are

interested

in

the

case

when

weights

are

arbitrary

real

numbers

-LRB-

It

turns

out

that

as

far

as

the

results

given

here

the

existence

of

just

one

irrational

weight

is

all

that

is

needed

-RRB-

The

Model

The

model

we

work

with

is

that

described

by

an

iteration

equation

such

as

-LRB-

-RRB-

For

notational

simplicity

we

often

summarize

this

equation

writing

x

-LRB-

t

-RRB-

instead

of

x

-LRB-

t

-RRB-

and

then

dropping

arguments

t

we

also

write

this

in

vector

form

as

x

oe

-LRB-

Ax

Bu

c

-RRB-

-LRB-

-RRB-

where

x

is

now

a

vector

of

size

N

number

of

processors

u

is

a

vector

of

size

M

number

of

inputs

c

is

an

N-vector

and

A

and

B

are

respectively

real

matrices

of

sizes

N

N

and

N

M

-LRB-

Now

oe

denotes

application

of

oe

into

each

coordinate

of

x.

-RRB-

Of

course

one

can

drop

the

vector

c

from

this

description

at

the

cost

of

adding

a

coordinate

but

it

is

often

useful

to

have

c

explicitly

and

this

allows

us

to

take

initial

states

to

be

x

which

corresponds

to

the

intuitive

idea

that

the

system

is

at

rest

before

the

first

input

appears

As

part

of

the

description

we

assume

that

we

have

singled

out

a

subset

of

the

N

processors

say

xip

these

are

the

p

output

processors

and

they

are

used

to

communicate

the

outputs

of

the

network

to

the

environment

Thus

a

net

is

specified

by

the

data

-LRB-

A

B

c

-RRB-

together

with

a

subset

of

its

nodes

In

our

further

development

both

input

and

output

channels

will

be

forced

to

carry

only

binary

data

Input

and

output

are

streams

that

is

one

input

letter

is

transferred

at

each

time

-LRB-

via

M

binary

lines

-RRB-

and

one

output

letter

is

produced

at

a

time

-LRB-

and

appears

in

the

output

via

p

binary

lines

-RRB-

As

opposed

to

the

I/O

the

computations

inside

the

network

will

in

general

involve

continuous

real

values

We

call

a

system

defined

by

equations

such

as

-LRB-

-RRB-

simply

a

network

or

processor

network

In

the

neural

network

literature

these

are

called

recurrent

first-order

neural

nets

We

show

later

that

considering

higher-order

nets

those

in

which

multiplications

of

activations

and/or

inputs

are

allowed

does

not

result

in

any

gain

in

computational

capabilities

-LRB-

up

to

polynomial

time

-RRB-

The

Finite

Structure

We

should

emphasize

from

the

outset

that

our

networks

are

built

up

of

finitely

many

processors

whose

number

does

not

increase

with

the

length

of

the

input

There

is

a

small

number

of

input

channels

-LRB-

just

two

in

our

main

result

-RRB-

into

which

inputs

get

presented

sequentially

We

assume

that

the

structure

of

the

network

including

the

values

of

the

interconnection

weights

does

not

change

in

time

but

rather

remains

constant

What

changes

in

time

are

the

activation

values

or

outputs

of

each

processor

which

are

used

in

the

next

iteration

-LRB-

A

synchronous

update

model

is

used

-RRB-

In

this

sense

our

model

is

very

uniform

in

contrast

with

certain

models

used

in

the

past

including

those

used

in

-LSB-

-RSB-

or

in

the

cellular

automata

literature

which

allow

the

number

of

units

to

increase

over

time

and

often

even

the

structure

to

change

depending

on

the

length

of

inputs

being

presented

The

Meaning

of

-LRB-

Non-Computable

-RRB-

Real

Weights

One

may

ask

about

the

meaning

of

real

weights

In

response

we

recall

that

our

intention

is

to

model

systems

in

which

certain

real

numbers

corresponding

to

values

of

resistances

capacitances

physical

constants

and

so

forth

may

not

be

directly

measurable

indeed

may

not

even

be

computable

real

numbers

but

they

affect

the

macroscopic

behavior

of

the

system

For

instance

imagine

a

spring/mass

system

The

dynamical

behavior

of

this

system

is

influenced

by

several

real

valued

constants

such

as

stiffness

and

friction

coefficients

On

any

finite

time

interval

one

could

replace

these

constants

by

rational

numbers

and

the

same

qualitative

behavior

is

observed

but

the

long-term

characteristics

of

the

system

depend

on

the

true

values

We

take

this

use

of

real

numbers

as

a

basic

feature

of

analog

computation

-LRB-

Another

characteristic

would

be

the

use

of

differential

as

opposed

to

difference

equations

but

technical

difficulties

make

that

further

study

harder

and

we

will

defer

it

to

future

work

-RRB-

What

is

interesting

is

to

find

a

class

of

such

systems

which

on

the

one

hand

is

rich

enough

to

exhibit

behavior

that

is

not

captured

by

digital

computation

while

still

being

amenable

to

useful

theoretical

analysis

and

in

particular

so

that

the

imposition

of

resource

constraints

results

in

nontrivial

reduction

of

computational

power

That

this

is

in

accordance

with

models

currently

used

in

neural

net

studies

is

especially

attractive

Previous

work

Many

authors

have

reported

successful

applications

when

using

neural

networks

for

various

computational

tasks

including

classification

and

optimization

problems

Special

purpose

analog

chips

are

being

built

to

implement

these

solutions

directly

in

hardware

see

for

instance

-LSB-

-RSB-

-LSB-

-RSB-

However

very

little

work

has

been

done

in

the

direction

of

exploring

the

ultimate

capabilities

of

such

devices

from

a

theoretical

standpoint

Part

of

the

problem

is

that

much

interesting

work

notwithstanding

analog

computation

is

hard

to

model

as

difficult

questions

about

precision

of

data

and

readout

of

results

are

immediately

encountered

see

for

instance

-LSB-

-RSB-

and

the

many

references

there

With

the

constraint

of

an

unchanging

structure

it

is

easy

to

see

that

classical

McCullochPitts

that

is

binary

neurons

would

have

no

more

power

than

finite

automata

which

is

not

an

interesting

situation

from

a

theoretical

complexity

point

of

view

Therefore

and

also

because

this

is

what

is

done

in

practical

applications

of

neural

nets

and

because

it

provides

a

closer

analogy

to

biological

systems

we

take

our

neurons

to

have

a

graded

analog

response

For

mathematical

tractability

we

pick

this

response

function

to

be

the

saturation

function

oe

defined

in

Equation

-LRB-

-RRB-

This

is

a

sigmoidal

nonlinearity

one

could

also

develop

a

theory

using

instead

of

oe

a

differentiable

function

such

as

-LRB-

e?x

-RRB-

but

this

presents

technical

difficulties

which

we

prefer

to

avoid

in

this

presentation

In

any

case

we

prove

below

that

no

gain

in

computational

abilities

even

in

polynomial

time

computation

can

be

achieved

by

using

more

general

sigmoids

as

long

as

they

satisfy

certain

reasonable

assumptions

It

is

important

to

note

that

graded

responses

as

opposed

to

a

threshold-binary

output

are

more

reasonable

in

models

of

computing

devices

as

it

is

not

reasonable

to

assume

that

physical

devices

can

discern

clearly

two

values

which

are

arbitrarily

close

This

continuity

in

behavior

is

a

basic

characteristic

of

our

model

In

the

paper

-LSB-

-RSB-

the

author

studies

a

class

of

machines

with

just

linear

activation

functions

and

shows

that

this

class

is

at

least

as

powerful

as

any

Turing

Machine

-LRB-

and

clearly

has

superTuring

capabilities

as

well

-RRB-

It

is

essential

in

that

model

however

that

the

number

of

neurons

be

allowed

to

be

infinite

as

a

matter

of

fact

in

-LSB-

-RSB-

the

number

of

such

units

is

even

uncountable

as

the

construction

relies

on

using

different

neurons

to

encode

different

possible

tape

configurations

in

Turing

Machines

The

work

closest

to

ours

seems

to

be

that

on

real-number-based

computation

started

by

Blum

Shub

and

Smale

-LRB-

see

e.g.

-LSB-

-RSB-

-RRB-

we

believe

that

our

setup

is

far

simpler

and

is

much

more

appropriate

if

one

is

interested

in

studying

neural

networks

or

distributed

processor

environments

In

the

related

previous

paper

-LSB-

-RSB-

there

were

different

models

for

each

input

size

the

model

allowed

for

no

loops

and

the

emphasis

was

on

comparisons

with

similar

models

made

up

of

binary

processors

The

remainder

of

this

paper

is

organized

as

follows

Section

includes

the

basic

definitions

of

networks

and

circuits

and

states

the

main

theorem

regarding

the

relationships

between

these

two

models

Sections

and

contain

the

proof

of

this

theorem

Section

shows

that

circuit

-LRB-

F

-LRB-

n

-RRB-

-RRB-

net

-LRB-

Poly

-LRB-

F

-LRB-

n

-RRB-

-RRB-

-RRB-

and

section

proves

that

net

-LRB-

F

-LRB-

n

-RRB-

-RRB-

circuit

-LRB-

Poly

-LRB-

F

-LRB-

n

-RRB-

-RRB-

-RRB-

Section

states

some

corollaries

for

neural

networks

which

follow

from

the

above

relation

with

circuits

We

also

define

there

a

notion

of

nondeterministic

network

In

section

we

briefly

compare

our

model

to

the

Blum

Shub

and

Smale

model

of

computation

over

the

reals

In

section

we

show

that

our

model

does

not

gain

power

if

one

lets

each

neuron

compute

a

polynomial

function

rather

than

just

affine

combinations

of

the

activations

of

all

the

neurons

and

the

external

inputs

or

by

allowing

more

general

activation

functions

than

the

piecewise

linear

one

We

conclude

in

section

with

a

discussion

on

analog

and

non-Turing

computation

We

now

turn

to

precise

definitions

Basic

Definitions

As

we

discussed

above

we

consider

synchronous

networks

which

can

be

represented

as

dynamical

systems

whose

state

at

each

instant

is

a

real

vector

x

-LRB-

t

-RRB-

IRN

The

ith

coordinate

of

this

vector

represents

the

activation

value

of

the

ith

processor

at

time

t

In

matrix

form

the

equations

are

as

in

-LRB-

-RRB-

for

suitable

matrices

A

B

and

vector

c.

Given

a

system

of

equations

such

as

-LRB-

-RRB-

an

initial

state

x

-LRB-

-RRB-

and

an

infinite

input

sequence

u

u

-LRB-

-RRB-

u

-LRB-

-RRB-

we

can

define

iteratively

the

state

x

-LRB-

t

-RRB-

at

time

t

for

each

integer

t

as

the

value

obtained

by

recursively

solving

the

equations

This

gives

rise

in

turn

to

a

sequence

of

output

values

by

restricting

attention

to

the

output

processors

we

refer

to

this

sequence

as

the

output

produced

by

the

input

u

starting

from

the

given

initial

state

Recognizing

Languages

To

define

what

we

mean

by

a

net

recognizing

a

language

L

we

must

first

define

a

formal

network

a

network

which

adheres

to

a

rigid

encoding

of

its

input

and

output

We

proceed

as

in

-LSB-

-RSB-

and

define

formal

nets

with

two

binary

input

lines

The

first

of

these

is

a

data

line

and

it

is

used

to

carry

a

binary

input

signal

when

no

signal

is

present

it

defaults

to

zero

The

second

is

the

validation

line

and

it

indicates

when

the

data

line

is

active

it

takes

the

value

while

the

input

is

present

there

and

thereafter

We

use

D

and

V

to

denote

the

contents

of

these

two

lines

respectively

so

u

-LRB-

t

-RRB-

-LRB-

D

-LRB-

t

-RRB-

V

-LRB-

t

-RRB-

-RRB-

for

each

t

We

always

take

the

initial

state

x

-LRB-

-RRB-

to

be

zero

and

to

be

an

equilibrium

state

that

is

oe

-LRB-

c

-RRB-

We

assume

that

there

are

two

output

processors

which

also

take

the

role

of

data

and

validation

lines

and

are

denoted

Od

-LRB-

t

-RRB-

Ov

-LRB-

t

-RRB-

respectively

-LRB-

The

convention

of

using

two

input

lines

allows

us

to

have

all

external

signals

be

binary

of

course

many

other

conventions

are

possible

and

would

give

rise

to

the

same

results

for

instance

one

could

use

a

three-valued

input

say

with

values

f

where

indicates

that

no

signal

is

present

and

are

the

two

possible

binary

input

values

-RRB-

We

now

encode

each

word

ff

ffk

as

follows

Let

uff

-LRB-

t

-RRB-

-LRB-

Vff

-LRB-

t

-RRB-

Dff

-LRB-

t

-RRB-

-RRB-

t

where

Vff

-LRB-

t

-RRB-

-LRB-

if

t

k

otherwise

and

Dff

-LRB-

t

-RRB-

-LRB-

ffk

if

t

k

otherwise

Given

a

formal

net

N

with

two

inputs

as

above

we

say

that

a

word

ff

is

classified

in

time

o

if

the

following

property

holds

the

output

sequence

y

-LRB-

t

-RRB-

-LRB-

Od

-LRB-

t

-RRB-

Ov

-LRB-

t

-RRB-

-RRB-

produced

by

uff

when

starting

from

x

-LRB-

-RRB-

has

the

form

Od

-LCB-

z

-RCB-

o

Ov

-LCB-

z

-RCB-

o

where

ff

or

Let

T

IN

IN

be

a

function

on

natural

numbers

We

say

that

the

language

L

is

recognized

in

time

T

by

the

formal

net

N

provided

that

each

word

ff

is

classified

in

time

o

T

-LRB-

jffj

-RRB-

and

ff

equals

when

ff

L

and

is

otherwise

Circuit

Families

We

briefly

recall

some

of

the

basic

definitions

of

non-uniform

families

of

circuits

A

Boolean

circuit

is

a

directed

acyclic

graph

Its

nodes

of

in-degree

are

called

input

nodes

while

the

rest

are

called

gates

and

are

labeled

by

one

of

the

Boolean

functions

AND

OR

or

NOT

-LRB-

the

first

two

seen

as

functions

of

many

variables

the

last

one

as

a

unary

function

-RRB-

One

of

the

nodes

which

has

no

outgoing

edges

is

designated

as

the

output

node

The

size

of

the

circuit

is

the

total

number

of

gates

Adding

if

necessary

extra

gates

we

assume

that

nodes

are

arranged

into

levels

d

where

the

input

nodes

are

at

level

zero

the

output

node

is

at

level

d

and

each

node

only

has

incoming

edges

from

the

previous

level

The

depth

of

the

circuit

is

d

and

its

width

is

the

maximum

size

of

each

level

Each

gate

computes

the

corresponding

Boolean

function

of

the

values

from

the

previous

level

and

the

value

obtained

is

considered

as

an

input

to

be

used

by

the

successive

level

in

this

fashion

each

circuit

computes

a

Boolean

function

of

the

inputs

A

family

of

circuits

C

is

a

set

of

circuits

fcn

n

INg

These

have

sizes

SC

-LRB-

n

-RRB-

depth

DC

-LRB-

n

-RRB-

and

width

WC

-LRB-

n

-RRB-

n

which

are

assumed

to

be

monotone

nondecreasing

functions

If

L

we

say

that

the

language

L

is

computed

by

the

family

C

if

the

characteristic

function

of

L

is

computed

by

cn

for

each

n

IN

The

qualifier

nonuniform

serves

as

a

reminder

that

there

is

no

requirement

that

circuit

families

be

recursively

described

It

is

this

lack

of

classical

computability

that

makes

circuits

a

possible

model

of

resource-bounded

computing

as

emphasized

in

-LSB-

-RSB-

We

will

show

that

recurrent

neural

networks

although

more

uniform

in

the

sense

that

they

have

an

unchanging

physical

structure

share

exactly

the

same

power

If

L

is

recognized

by

the

formal

net

N

in

time

T

we

write

OEN

L

and

TN

T

If

L

is

computed

by

the

family

of

circuits

C

we

write

OEC

L

We

are

interested

in

comparing

the

functions

TN

and

SC

for

formal

nets

and

circuits

so

that

OEN

OEC

Statement

Of

Result

Recall

that

net

-LRB-

T

-LRB-

n

-RRB-

-RRB-

is

the

class

of

languages

recognized

by

formal

networks

-LRB-

with

real

weights

-RRB-

in

time

T

-LRB-

n

-RRB-

and

that

circuit

-LRB-

S

-LRB-

n

-RRB-

-RRB-

is

the

class

of

languages

recognized

by

-LRB-

non-uniform

-RRB-

families

of

circuits

of

size

S

-LRB-

n

-RRB-

Theorem

Let

F

be

so

that

F

-LRB-

n

-RRB-

n.

Then

net

-LRB-

F

-LRB-

n

-RRB-

-RRB-

circuit

-LRB-

Poly

-LRB-

F

-LRB-

n

-RRB-

-RRB-

-RRB-

and

circuit

-LRB-

F

-LRB-

n

-RRB-

-RRB-

net

-LRB-

Poly

-LRB-

F

-LRB-

n

-RRB-

-RRB-

-RRB-

More

precisely

we

prove

the

following

two

facts

For

each

function

F

-LRB-

n

-RRB-

n

ffl

circuit

-LRB-

F

-LRB-

n

-RRB-

-RRB-

net

-LRB-

nF

-LRB-

n

-RRB-

-RRB-

ffl

net

-LRB-

F

-LRB-

n

-RRB-

-RRB-

circuit

-LRB-

F

-LRB-

n

-RRB-

-RRB-

Circuit

Families

Are

Simulated

By

Networks

We

start

by

reducing

circuit

families

to

networks

The

proof

will

construct

a

fixed

universal

net

having

roughly

N

processors

which

through

the

setting

of

a

particular

real

weight

which

encodes

an

entire

circuit

family

can

simulate

that

family

Theorem

There

exists

a

positive

integer

N

such

that

the

following

property

holds

For

each

circuit

family

C

of

size

SC

-LRB-

n

-RRB-

there

exists

an

N-processor

formal

network

N

N

-LRB-

C

-RRB-

so

that

OEN

OEC

and

TN

-LRB-

n

-RRB-

O

-LRB-

-LRB-

n

-RRB-

-RRB-

The

proof

is

provided

in

the

remainder

of

this

section

The

circuit

Encoding

Given

a

circuit

c

with

size

s

width

w

and

wi

gates

in

the

ith

level

we

encode

it

as

a

finite

sequence

over

the

alphabet

as

follows

ffl

The

encoding

of

each

level

i

starts

with

the

letter

Levels

are

encoded

successively

starting

with

the

bottom

level

and

ending

with

the

top

one

ffl

At

each

level

gates

are

encoded

successively

The

encoding

of

a

gate

g

consists

of

three

parts

a

starting

symbol

a

code

for

the

gate

type

and

a

code

to

indicate

which

gate

feeds

into

it

-LCB-

It

starts

with

the

letter

-LCB-

A

two

digit

sequence

denotes

the

type

of

the

gate

fAND

OR

NOTg

respectively

-LCB-

If

gate

g

is

in

level

i

then

the

input

to

g

is

represented

as

a

sequence

in

such

that

the

jth

position

in

the

sequence

is

if

and

only

if

the

jth

gate

of

the

-LRB-

i

-RRB-

th

level

feeds

into

gate

g

The

encoding

of

a

gate

g

in

level

i

is

of

length

-LRB-

wi

-RRB-

The

length

of

the

encoding

of

a

circuit

c

is

l

-LRB-

c

-RRB-

jen

-LRB-

c

-RRB-

j

O

-LRB-

sw

-RRB-

Example

The

circuit

in

Figure

is

encoded

as

en

-LSB-

-RSB-

-LCB-

z

-RCB-

-LCB-

z

-RCB-

-LCB-

z

-RCB-

-LCB-

z

-RCB-

For

instance

the

NOT

gate

corresponds

to

the

subsequence

It

starts

with

the

letter

followed

by

the

two

digits

denoting

that

the

gate

is

of

type

NOT

and

ends

with

which

indicates

that

only

the

second

input

feeds

into

the

gate

We

encode

a

non-uniform

family

of

circuits

C

of

size

S

-LRB-

n

-RRB-

as

an

infinite

sequence

e

-LRB-

C

-RRB-

en

-LSB-

-RSB-

en

-LSB-

-RSB-

en

-LSB-

-RSB-

-LRB-

-RRB-

where

en

-LSB-

ci

-RSB-

is

the

encoding

of

ci

in

the

reversed

order

AAAAK

S

S

S

So

Z

Z

Z

Z

Z

-RCB-

@I

Z

Z

Z

Z

-RCB-

aeaeaeae

Or

Or

NotAnd

Figure

Circuit

Figure

Values

of

the

circuit

encoding

Let

b

be

a

natural

number

and

r

a

finite

or

infinite

sequence

of

natural

numbers

smaller

than

b

The

interpretation

of

the

sequence

r

in

base

b

is

the

number

rjb

X

i

ri

bi

Generally

two

different

sequences

may

result

in

the

same

encoding

For

instance

both

r

and

r

provide

However

restricted

to

the

sequences

we

will

consider

the

encoding

is

one-to-one

We

can

interpret

formula

-LRB-

-RRB-

in

base

We

denote

this

representation

of

the

family

of

circuits

C

as

C

C

en

-LSB-

-RSB-

en

-LSB-

-RSB-

en

-LSB-

-RSB-

-LRB-

-RRB-

Let

ci

be

the

ith

circuit

in

the

family

We

denote

by

c

en

-LSB-

ci

-RSB-

the

encoding

en

-LSB-

ci

-RSB-

interpreted

in

base

Cantor

Like

Set

Encoding

A

number

which

encodes

a

family

of

circuits

or

one

which

is

a

suffix

of

such

an

encoding

is

a

number

between

and

However

not

every

number

in

the

range

-LSB-

-RSB-

can

appear

in

this

manner

If

the

first

digit

to

the

right

of

the

decimal

point

is

then

the

value

of

the

encoding

ranges

in

-LSB-

-RSB-

if

it

is

the

value

ranges

in

-LSB-

-RSB-

and

so

forth

The

number

can

not

lie

in

any

of

the

ranges

-LSB-

-RSB-

for

i

The

second

digit

after

the

decimal

point

decides

the

possible

range

relative

to

the

currently

candidate

range

see

Figure

In

summary

not

every

value

in

-LSB-

-RSB-

appears

The

set

of

possible

values

is

not

continuous

and

has

holes

Such

a

set

of

values

with

holes

is

a

Cantor

set

Its

self-similar

structure

means

that

bit

-LRB-

base

-RRB-

shifts

preserve

the

holes

The

advantage

of

this

approach

is

that

there

is

never

a

need

to

distinguish

among

two

very

close

numbers

in

order

to

read

the

desired

circuit

out

of

the

encoding

the

circuit

can

be

then

retrieved

with

finite-precision

operations

employing

a

finite

number

of

neurons

A

Circuit

Retrieval

Lemma

For

each

-LRB-

non-uniform

-RRB-

family

of

circuits

C

there

exists

a

network

NR

-LRB-

C

-RRB-

with

one

input

line

such

that

starting

from

the

zero

initial

state

and

given

the

input

signal

u

-LRB-

-RRB-

-LCB-

z

-RCB-

n

n

u

-LRB-

t

-RRB-

for

t

NR

-LRB-

C

-RRB-

outputs

xr

-LCB-

z

-RCB-

Pni

l

-LRB-

ci

-RRB-

c

en

-LSB-

cn

-RSB-

Proof

Let

Denote

by

the

Cantor

which

consists

of

all

those

real

numbers

q

which

admit

an

expansion

of

the

form

q

X

i

ffi

-LRB-

-RRB-

with

each

ffi

Let

IR

-LSB-

-RSB-

be

the

function

-LSB-

x

-RSB-

if

x

if

x

if

x

-LRB-

-RRB-

Let

IR

-LSB-

-RSB-

be

the

function

-LSB-

x

-RSB-

if

x

c

if

x

if

x

-LRB-

-RRB-

Note

that

for

each

q

X

i

ffi

we

may

think

of

-LSB-

q

-RSB-

as

the

select

left

operation

since

-LSB-

q

-RSB-

and

of

-LSB-

q

-RSB-

as

the

shift

left

operation

since

-LSB-

q

-RSB-

X

i

ffi

For

each

i

q

-LSB-

i

-LSB-

q

-RSB-

-RSB-

ffi

The

following

procedure

summarizes

the

task

to

be

performed

by

the

network

constructed

below

which

in

turn

satisfies

the

requirements

of

the

lemma

Procedure

Retrieval

-LRB-

C

n

-RRB-

Variables

counter

y

z

Begin

counter

y

z

C

While

counter

n

Parbegin

z

-LSB-

z

-RSB-

if

-LSB-

z

-RSB-

then

increment

counter

Parend

While

-LSB-

z

-RSB-

Parbegin

z

-LSB-

z

-RSB-

y

-LRB-

y

-LSB-

z

-RSB-

-RRB-

Parend

Return

-LRB-

y

-RRB-

End

The

functions

and

can

not

be

programmed

with

the

neural

network

model

due

to

their

discontinuity

However

we

can

program

the

functions

which

coincide

with

respectively

on

-LSB-

q

-RSB-

j

-LRB-

-RRB-

joe

-LRB-

j

-RRB-

-LRB-

-RRB-

and

-LSB-

q

-RSB-

j

oe

-LRB-

-LRB-

-RRB-

-RRB-

-LRB-

-RRB-

Figure

the

function

-LSB-

x

-RSB-

Figure

the

function

-LSB-

x

-RSB-

The

retrieval

prodecure

is

then

achieved

by

the

following

network

x

i

oe

-LRB-

i

-RRB-

i

x

oe

-LRB-

-RRB-

x

oe

-LRB-

-RRB-

x

oe

-LRB-

-LRB-

-RRB-

-RRB-

x

oe

-LRB-

-RRB-

x

oe

-LRB-

u

-RRB-

x

oe

-LRB-

-RRB-

x

oe

-LRB-

-RRB-

x

oe

-LRB-

-RRB-

If

the

input

u

arrives

at

time

then

-LRB-

-RRB-

k

-LSB-

C

-RSB-

-LRB-

because

of

equation

-LRB-

-RRB-

-RRB-

Processors

serve

to

implement

the

counter

and

processor

is

the

output

processor

This

network

satisfies

the

requirements

of

the

lemma

Circuit

Simulation

By

A

Network

Let

ff

be

a

binary

sequence

Denote

by

en

-LSB-

ff

-RSB-

the

sequence

that

substitutes

-LRB-

-RRB-

for

each

ffi

and

by

c

en

-LSB-

ff

-RSB-

the

interpretation

of

en

-LSB-

ff

-RSB-

in

base

that

is

en

-LSB-

ff

-RSB-

We

next

construct

a

universal

net

for

interpreting

circuits

Lemma

There

exists

a

network

Ns

such

that

for

each

circuit

c

and

binary

sequence

ff

starting

from

the

zero

initial

state

and

applying

the

input

signal

c

en

-LSB-

c

-RSB-

c

en

-LSB-

ff

-RSB-

Ns

outputs

-LCB-

z

-RCB-

T

y

xv

-LCB-

z

-RCB-

T

where

y

is

the

response

of

circuit

c

on

the

input

ff

and

T

O

-LRB-

l

-LRB-

c

-RRB-

jffj

-RRB-

Proof

It

is

easy

to

verify

that

given

any

circuit

there

is

a

three-tape

Turing

Machine

which

can

simulate

the

given

circuit

in

time

O

-LRB-

l

-LRB-

c

-RRB-

jffj

-RRB-

This

Turing

Machine

would

employ

its

tapes

to

store

the

circuit

encoding

the

input

and

output

encoding

and

the

current

level

's

calculation

respectively

Now

we

can

simulate

this

machine

by

a

net

Indeed

we

proved

in

-LRB-

-LSB-

-RSB-

-RRB-

that

if

M

is

a

k-tape

Turing

Machine

with

s

states

which

computes

in

time

T

a

function

f

on

binary

input

strings

then

there

exists

a

rational

network

N

which

consists

of

s

processors

that

computes

the

same

function

f

in

time

O

-LRB-

T

-RRB-

Closer

counting

shows

that

less

than

processors

suffice

Remark

If

the

lemma

would

only

require

an

estimate

of

a

polynomial

number

of

processors

as

opposed

to

the

more

precise

estimate

that

we

obtain

the

proof

would

have

been

immediate

from

the

consideration

of

the

circuit

value

problem

-LRB-

CVP

-RRB-

This

is

the

problem

of

recognizing

the

set

of

all

pairs

x

y

where

x

and

y

encodes

a

circuit

with

jxj

input

lines

which

outputs

on

input

x

It

is

known

that

CVP

P

-LRB-

-LSB-

-RSB-

volume

I

pg

-RRB-

Proof

Circuit

Families

Are

Simulated

By

Networks

Proof

of

Theorem

Let

C

be

a

circuit

family

We

construct

the

required

formal

network

as

a

composition

of

the

following

three

networks

ffl

An

input

network

NI

which

receives

the

input

ff

-LCB-

z

-RCB-

jffj

and

computes

c

en

-LSB-

ff

-RSB-

and

for

each

ff

This

network

is

trivial

to

implement

ffl

A

retrieval

network

NR

-LRB-

c

-RRB-

as

described

in

Lemma

which

receives

from

NI

and

computes

c

en

-LSB-

cjffj

-RSB-

-LRB-

Note

that

during

the

encoding

operation

network

NI

produces

an

output

of

zero

and

NR

-LRB-

c

-RRB-

remains

in

its

initial

state

-RRB-

ffl

A

simulation

network

NS

as

stated

in

Lemma

which

receives

c

en

-LSB-

cjffj

-RSB-

and

c

en

-LSB-

ffj

and

computes

-LCB-

z

-RCB-

T

OEc

-LRB-

ff

-RRB-

xv

-LCB-

z

-RCB-

T

Notice

that

out

of

the

above

three

networks

only

NR

depends

on

the

specific

family

of

circuits

C.

Moreover

all

weights

can

be

taken

to

be

rational

numbers

except

for

the

one

weight

that

encodes

the

entire

circuit

family

The

time

complexity

to

compute

the

response

of

C

to

the

input

ff

is

dominated

by

that

of

retrieving

the

circuit

description

Thus

the

complexity

is

of

order

T

O

jffj

X

i

l

-LRB-

ci

-RRB-

A

We

remarked

that

the

length

of

the

encoding

l

-LRB-

ci

-RRB-

is

of

order

O

-LRB-

WC

-LRB-

i

-RRB-

SC

-LRB-

i

-RRB-

-RRB-

which

is

itself

O

-LRB-

-LRB-

i

-RRB-

-RRB-

Since

SC

-LRB-

i

-RRB-

SC

-LRB-

i

-RRB-

for

i

we

achieve

the

claimed

bound

T

O

-LRB-

jffj

-LRB-

jffj

-RRB-

-RRB-

Remark

In

case

of

bounded

fan-in

the

standard

encoding

of

circuit

cn

is

of

length

l

-LRB-

cn

-RRB-

O

-LRB-

SC

-LRB-

n

-RRB-

log

-LRB-

SC

-LRB-

n

-RRB-

-RRB-

-RRB-

The

total

running

time

of

the

algorithm

is

then

O

-LRB-

nSC

-LRB-

n

-RRB-

log

-LRB-

SC

-LRB-

n

-RRB-

-RRB-

-RRB-

Networks

Are

Simulated

By

Circuit

Families

We

next

state

the

reverse

simulation

of

nets

by

nonuniform

families

of

circuits

Theorem

Let

N

be

a

formal

network

that

computes

in

time

T

IN

IN

There

exists

a

nonuniform

family

of

circuits

C

-LRB-

N

-RRB-

of

size

O

-LRB-

T

-RRB-

depth

O

-LRB-

T

log

-LRB-

T

-RRB-

-RRB-

and

width

O

-LRB-

T

-RRB-

that

accepts

the

same

language

as

N

does

The

proof

is

given

in

the

next

two

subsections

In

the

first

part

we

replace

a

single

formal

network

by

a

family

of

formal

networks

with

small

rational

weights

-LRB-

This

is

unrelated

to

the

standard

fact

for

threshold

gates

that

weights

can

be

taken

to

have

n

log

n

bits

-RRB-

In

the

second

part

we

simulate

such

a

family

of

formal

networks

by

circuits

Linear

Precision

Suffices

Define

a

processor

to

be

a

designated

output

processor

if

its

activation

value

is

used

as

an

output

of

the

network

-LRB-

i.e.

it

is

an

output

processor

-RRB-

and

is

not

fed

into

any

other

processor

A

formal

network

for

which

its

two

output

processors

are

designated

is

called

an

output

designated

network

Its

processors

which

are

not

the

designated

output

processors

are

called

internal

processors

For

the

next

result

we

introduce

the

notion

of

a

q-truncation

net

This

is

a

processor

network

in

which

the

update

equations

take

the

form

x

i

q-Truncation

-LSB-

oe

-LRB-

NX

j

aijxj

M

X

j

bijuj

ci

-RRB-

-RSB-

where

q-Truncation

means

the

operation

of

truncating

after

q

bits

Lemma

Let

N

be

an

output

designated

network

If

N

computes

in

time

T

there

exists

a

family

of

T

-LRB-

n

-RRB-

Truncation

output

designated

networks

-LRB-

n

-RRB-

such

that

ffl

For

each

n

-LRB-

n

-RRB-

has

the

same

number

of

processors

and

input

and

output

channels

as

N

does

ffl

The

weights

feeding

into

the

internal

processors

of

-LRB-

n

-RRB-

are

like

those

of

N

but

truncated

after

O

-LRB-

T

-LRB-

n

-RRB-

-RRB-

bits

ffl

For

each

designated

output

processor

in

N

if

this

processor

computes

x

i

oe

-LRB-

f

-RRB-

where

f

is

a

linear

function

of

processors

and

inputs

then

the

respective

processor

in

-LRB-

n

-RRB-

computes

oe

-LRB-

f

-RRB-

where

f

is

the

same

as

the

linear

function

f

but

applied

instead

to

the

processors

of

-LRB-

n

-RRB-

and

with

weights

truncated

at

O

-LRB-

T

-LRB-

n

-RRB-

-RRB-

bits

ffl

The

respective

output

processors

of

N

and

-LRB-

n

-RRB-

have

the

same

activation

values

at

all

times

t

T

-LRB-

n

-RRB-

Proof

We

first

measure

the

difference

-LRB-

error

-RRB-

between

the

activations

of

the

corresponding

internal

processors

of

-LRB-

n

-RRB-

and

N

at

time

t

T

-LRB-

n

-RRB-

This

calculation

is

analogous

to

that

of

the

chop

error

in

floating

point

computation

-LSB-

-RSB-

We

use

the

following

notations

N

is

the

number

of

processors

M

is

the

number

of

input

lines

L

N

M

W

is

the

largest

absolute

value

of

the

weights

of

N

W

W

xi

-LRB-

t

-RRB-

is

the

value

of

processor

i

of

network

N

at

time

t.

ffiw

-LRB-

-RRB-

and

ffip

are

the

truncation

errors

at

weights

and

processors

respectively

fflt

is

the

largest

accumulated

error

at

time

t

in

processors

of

-LRB-

n

-RRB-

u

is

the

input

to

both

N

and

-LRB-

n

-RRB-

-LRB-

u

-LRB-

t

-RRB-

M

for

t

n.

-RRB-

aij

bij

and

ci

are

the

weights

influencing

processor

i

of

network

N

xi

-LRB-

t

-RRB-

aij

bij

and

ci

are

the

respective

activation

values

of

processors

and

weights

of

-LRB-

n

-RRB-

Network

-LRB-

n

-RRB-

computes

at

each

step

x

i

q-Truncation

-LSB-

oe

-LRB-

NX

j

aij

xj

M

X

i

bij

uj

ci

-RRB-

-RSB-

We

assume

inductively

on

t

that

for

all

internal

processors

i

j

j

xi

-LRB-

t

-RRB-

xi

-LRB-

t

-RRB-

j

fflt

j

aij

-LRB-

t

-RRB-

aij

-LRB-

t

-RRB-

j

ffiw

j

bij

-LRB-

t

-RRB-

bij

-LRB-

t

-RRB-

j

ffiw

and

j

ci

-LRB-

t

-RRB-

ci

-LRB-

t

-RRB-

j

ffiw

Using

the

global

Lipschitz

property

joe

-LRB-

a

-RRB-

oe

-LRB-

b

-RRB-

j

ja

bj

it

follows

that

fflt

N

-LRB-

W

ffiw

-RRB-

fflt

-LRB-

N

M

-RRB-

ffiw

ffip

LW

fflt

Lffiw

ffip

Therefore

fflt

t

X

i

-LRB-

LW

-RRB-

i

-LRB-

Lffiw

ffip

-RRB-

-LRB-

LW

-RRB-

t

-LRB-

Lffiw

ffip

-RRB-

We

now

analyze

the

behavior

of

the

output

processors

We

need

to

prove

that

oe

-LRB-

f

-RRB-

when

oe

-LRB-

f

-RRB-

respectively

That

is

f

=-RRB-

f

and

f

=-RRB-

f

This

happens

if

jf

f

j

Arguing

as

earlier

the

condition

fflt

suffices

This

is

translated

into

the

requirement

-LRB-

Lffiw

ffip

-RRB-

-LRB-

LW

-RRB-

t

If

both

ffiw

and

ffip

are

bounded

by

-LRB-

LW

-RRB-

-LRB-

t

-RRB-

this

inequality

holds

This

happens

when

the

weights

and

the

processor

activations

are

truncated

after

O

-LRB-

t

log

-LRB-

LW

-RRB-

-RRB-

bits

As

L

and

W

are

constants

we

conclude

as

desired

that

a

sufficient

truncation

for

a

computation

of

length

T

is

O

-LRB-

T

-RRB-

The

Network

Simulation

by

a

Circuit

Lemma

Let

be

a

family

of

T

-LRB-

n

-RRB-

Truncation

output

designated

networks

where

all

networks

-LRB-

n

-RRB-

consist

of

N

processors

and

the

weights

are

all

rational

numbers

with

O

-LRB-

T

-RRB-

bits

Then

there

exists

a

circuit

family

C

of

size

O

-LRB-

T

-RRB-

depth

O

-LRB-

T

log

-LRB-

T

-RRB-

-RRB-

and

width

O

-LRB-

T

-RRB-

so

that

cn

accepts

the

same

language

as

-LRB-

n

-RRB-

does

on

Proof

We

sketch

the

construction

of

the

circuit

cn

which

corresponds

to

the

T

-LRB-

n

-RRB-

Truncation

output

designated

net

-LRB-

n

-RRB-

The

network

-LRB-

n

-RRB-

has

two

input

lines

data

and

validation

where

the

validation

line

sees

n

consecutive

's

followed

by

's

We

think

of

the

n

data

bits

on

the

data

line

which

appear

simultaneously

with

the

's

in

the

validation

line

as

data

input

of

size

n

These

n

bits

are

fed

simultaneously

into

cn

via

n

input

nodes

To

simulate

the

sequential

input

in

-LRB-

n

-RRB-

we

construct

an

input-subcircuit

which

preserves

the

input

as

it

is

to

be

released

one

bit

at

a

time

in

later

times

of

the

computation

The

input

subcircuit

is

of

size

nDC

-LRB-

n

-RRB-

Let

p

p

N

be

a

processor

of

-LRB-

n

-RRB-

We

associate

with

each

processor

p

a

subcircuit

sc

-LRB-

p

-RRB-

Each

processor

p

-LRB-

n

-RRB-

computes

a

truncated

sum

of

up

to

N

numbers

N

of

which

are

multiplications

of

two

T

bit

numbers

Hardwiring

the

weights

we

can

say

that

each

processor

computes

a

sum

of

-LRB-

TN

-RRB-

-LRB-

-RRB-

bit

numbers

Using

the

carry-look-ahead

method

-LSB-

-RSB-

the

summation

can

be

computed

via

a

subcircuit

of

depth

O

-LRB-

log

-LRB-

TN

-RRB-

-RRB-

width

O

-LRB-

T

-RRB-

and

size

O

-LRB-

T

-RRB-

-LRB-

This

depth

is

of

the

same

order

as

the

lower

bound

of

similar

tasks

see

-LSB-

-RSB-

-LSB-

-RSB-

-RRB-

As

for

the

saturation

one

gate

pu

is

sufficient

for

the

integer

part

As

only

O

-LRB-

T

-RRB-

bits

are

preserved

the

activation

of

each

processor

can

be

represented

in

binary

by

the

unit

gate

pu

and

the

most

significant

gates

pi

i

O

-LRB-

T

-RRB-

after

the

operation

AND

-LRB-

pi

-LRB-

pu

-RRB-

-RRB-

i

O

-LRB-

T

-RRB-

Let

sc

-LRB-

-RRB-

be

a

subcircuit

of

largest

depth

Pad

the

other

sc

-LRB-

p

-RRB-

's

with

demi

gates

-LRB-

e.g.

an

AND

gate

of

a

single

input

-RRB-

so

that

all

sc

-LRB-

p

-RRB-

's

are

of

equal

depth

The

output

of

circuit

sc

-LRB-

p

-RRB-

is

called

the

activation

of

sc

-LRB-

p

-RRB-

We

place

the

N

subcircuits

sc

-LRB-

p

-RRB-

p

N

to

compute

in

parallel

We

call

this

subcircuit

a

layer

A

layer

corresponds

to

one

step

in

the

computation

of

-LRB-

n

-RRB-

As

-LRB-

n

-RRB-

computes

in

time

T

-LRB-

n

-RRB-

T

-LRB-

n

-RRB-

layers

are

connected

sequentially

Each

layer

i

receives

the

ith

input

bit

from

the

input-subcircuit

and

the

N

activation

values

of

its

preceding

layer

-LRB-

except

for

layer

which

receives

input

only

-RRB-

This

main

subarchitecture

is

of

size

O

-LRB-

T

-RRB-

depth

O

-LRB-

T

log

-LRB-

T

-RRB-

-RRB-

and

width

O

-LRB-

T

-RRB-

where

T

T

-LRB-

n

-RRB-

As

-LRB-

n

-RRB-

may

compute

the

response

to

different

strings

of

size

n

in

different

times

of

order

O

-LRB-

T

-RRB-

we

construct

an

output-subcircuit

which

forces

the

response

to

every

string

of

size

n

to

appear

at

the

top

of

the

circuit

For

each

layer

i

T

we

apply

the

AND

function

to

the

output

of

the

subcircuits

sc

-LRB-

-RRB-

sc

-LRB-

-RRB-

where

are

the

output

processors

of

-LRB-

n

-RRB-

We

transfer

these

values

and

apply

the

OR

functions

to

them

The

resulting

value

is

the

output

of

the

circuit

When

OR

is

applied

at

each

layer

only

DC

-LRB-

n

-RRB-

gates

are

needed

for

this

subcircuit

The

resources

of

the

total

circuit

are

dominated

by

those

of

the

main

subarchitecture

The

proof

of

Theorem

follows

immediately

from

Lemma

and

Lemma

Corollaries

Let

net-p

and

net-exp

be

the

classes

of

languages

accepted

by

formal

networks

in

polynomial

time

and

exponential

time

respectively

Let

circuit-p

and

circuit-exp

be

the

classes

of

languages

accepted

by

families

of

circuits

in

polynomial

and

exponential

size

respectively

Corollary

net-p

circuit-p

and

net-exp

circuit-exp

The

class

circuit-p

is

often

called

P/poly

and

coincides

with

the

class

of

languages

recognized

by

Turing

Machine

with

advice

sequences

in

polynomial

time

The

following

corollary

states

that

this

class

also

coincides

with

the

class

of

languages

recognized

in

polynomial

time

by

Turing

Machines

that

consult

oracles

where

the

oracles

are

sparse

sets

A

sparse

set

S

is

a

set

in

which

for

each

length

n

the

number

of

words

in

S

of

length

at

most

n

is

bounded

by

some

polynomial

function

For

instance

any

tally

set

that

is

a

subset

of

is

an

example

of

a

sparse

set

The

class

P

-LRB-

S

-RRB-

for

a

given

sparse

set

S

is

the

class

of

all

languages

computed

by

Turing

machines

in

polynomial

time

and

using

queries

from

the

oracle

S.

From

-LSB-

-RSB-

volume

I

Theorem

pg

and

Corollary

we

conclude

as

follows

Corollary

net-p

-LSB-

S

sparse

P

-LRB-

S

-RRB-

From

-LSB-

-RSB-

volume

I

Theorem

pg

-LRB-

originally

-LSB-

-RSB-

-RRB-

we

conclude

as

follows

Corollary

net-exp

includes

all

possible

binary

languages

Furthermore

most

Boolean

functions

require

exponential

time

complexity

Nondeterministic

Neural

Networks

The

concept

of

a

nondeterministic

circuit

family

is

usually

defined

by

means

of

an

extra

input

whose

role

is

that

of

an

oracle

Similarly

we

define

a

nondeterministic

network

to

be

a

network

having

an

extra

binary

input

line

the

Guess

line

in

addition

to

the

Data

and

Validation

lines

A

language

L

is

said

to

accepted

by

a

nondeterministic

formal

network

N

in

time

B

if

L

fff

j

a

guess

OEN

-LRB-

ff

-RRB-

TN

-LRB-

ff

-RRB-

B

-LRB-

jffj

-RRB-

g

It

is

easy

to

see

that

Corollary

-LRB-

-RRB-

stated

for

the

deterministic

case

holds

for

the

nondeterministic

case

as

well

That

is

if

we

define

net-np

to

be

the

class

of

languages

accepted

by

nondeterministic

formal

networks

in

polynomial

time

and

circuit-np

to

be

the

class

of

languages

accepted

by

nondeterministic

non-uniform

families

of

circuits

of

polynomial

size

then

Corollary

net-np

circuit-np

Since

np

net-np

-LRB-

one

may

simulate

a

nondeterministic

Turing

Machine

by

a

nondeterministic

network

with

rational

weights

-RRB-

the

equality

net-np

net-p

implies

np

circuit-p

P/poly

Thus

from

-LSB-

-RSB-

we

conclude

Theorem

If

net-np

net-p

then

the

polynomial

hierarchy

collapses

to

The

above

result

says

that

a

theory

of

computation

similar

to

that

which

arises

in

the

classical

case

of

Turing

machine

computation

is

also

possible

for

our

model

of

analog

computation

In

particular

even

though

the

two

models

have

very

different

power

the

question

of

knowing

if

the

verification

of

solutions

to

problems

is

really

easier

than

finding

solutions

at

the

core

of

modern

computational

complexity

has

a

precise

corresponding

version

in

our

setup

and

its

solution

will

be

closely

related

to

that

of

the

classical

case

Of

course

it

follows

from

this

that

it

is

quite

likely

that

net-np

is

strictly

more

powerful

than

net-p

Complexity

Over

The

Reals

Blum

Shub

and

Smale

introduced

in

-LSB-

-RSB-

a

powerful

model

of

computation

over

the

real

numbers

This

model

allows

one

possible

formalization

of

the

notion

of

analog

computing

Three

main

characteristics

differentiate

our

neural

network

model

from

the

BSS

model

namely

ffl

The

BSS

model

allows

real-valued

inputs

rather

than

only

binary

ffl

In

the

BSS

model

values

can

be

compared

for

exact

equality

to

any

particular

value

e.g.

zero

That

is

exact

precision

is

available

This

is

not

possible

in

our

model

as

discontinuities

are

not

allowed

By

appropriate

choice

of

weights

we

are

able

however

to

distinguish

for

any

fixed

ffl

between

any

two

values

x

and

y

so

that

jy

xj

ffl

ffl

The

BSS

model

allows

for

an

infinite

range

of

values

in

registers

-LCB-

which

correspond

to

our

neurons

-LCB-

whereas

our

model

restricts

the

possible

range

of

values

to

an

adjustable

but

finite

bound

The

BSS

model

is

closely

related

to

the

model

that

is

obtained

if

two

types

of

neurons

are

available

Heaviside

neurons

that

compute

linear

threshold

functions

and

identity

neurons

This

model

allows

for

discontinuous

branching

as

in

the

BSS

model

Equivalence

of

Different

Dynamical

Systems

We

consider

dynamical

systems

-LCB-

which

we

will

call

generalized

processor

networks

-LCB-

with

far

less

restrictive

structure

than

the

recurrent

neural

network

model

which

was

described

in

equation

-LRB-

-RRB-

We

show

that

these

networks

are

not

more

powerful

up

to

polynomial

time

slowdown

than

the

previously

considered

model

Let

N

M

p

be

natural

numbers

A

generalized

processor

network

is

a

dynamical

system

that

consists

of

N

processors

xN

and

receives

its

input

-LRB-

t

-RRB-

-LRB-

t

-RRB-

uM

-LRB-

t

-RRB-

via

M

input

lines

A

subset

of

the

N

processors

say

xip

is

the

set

of

output

processors

of

the

system

used

to

communicate

the

output

of

the

system

to

the

environment

In

vector

form

a

generalized

processor

network

D

updates

its

processors

via

the

dynamic

equation

x

f

-LRB-

x

u

-RRB-

where

x

is

the

current

state

of

the

network

-LRB-

a

vector

-RRB-

u

is

an

external

input

-LRB-

also

possibly

a

vector

-RRB-

and

f

is

a

composition

of

functions

f

ffi

ss

where

ss

IRN+M

IRN

is

some

vector

polynomial

in

N

M

variables

with

real

coefficients

and

IRN

IRN

is

any

vector

function

which

has

a

bounded

range

and

is

locally

Lipschitz

-LRB-

Thus

the

composite

function

f

ffi

ss

again

satisfies

the

same

properties

-RRB-

We

also

assume

as

part

of

the

definition

of

generalized

processor

network

that

at

least

for

binary

inputs

of

the

type

considered

in

the

definition

of

formal

networks

given

in

section

D

outputs

soft

binary

information

That

is

there

exist

two

constants

ff

satisfying

ff

and

called

the

decision

thresholds

so

that

each

output

neuron

of

D

outputs

a

stream

of

numbers

each

of

which

is

either

smaller

than

ff

or

larger

than

We

interpret

the

outputs

of

each

output

neuron

y

as

a

binary

value

binary

-LRB-

y

-RRB-

-LRB-

if

y

ff

if

y

In

the

usual

model

we

studied

earlier

the

values

are

always

binary

but

we

allow

more

generality

to

show

that

even

if

one

allows

more

general

analog

values

no

increase

in

computational

power

is

attained

at

least

up

to

polynomial

time

Remark

The

above

assumptions

imply

that

for

each

ae

there

exists

a

constant

C

such

that

for

all

x

and

x

satisfying

that

jx

xj

ae

and

x

Range

-LRB-

-RRB-

-LRB-

the

absolute

value

sign

indicates

Euclidean

norm

-RRB-

the

following

property

holds

j

-LRB-

x

u

-RRB-

-LRB-

x

u

-RRB-

j

Cjx

xj

for

any

binary

vector

u

A

similar

property

holds

for

f

A

neural

network

is

a

special

case

of

a

generalized

processor

network

in

which

all

coordinates

of

the

function

compute

the

same

sigmoidal

function

and

the

polynomial

ss

is

a

first

order

polynomial

that

is

an

affine

function

For

the

next

result

we

need

to

generalize

the

notion

of

a

truncation

network

introduced

in

section

Let

D

be

a

generalized

processor

network

D

x

-LRB-

ss

-LRB-

x

u

-RRB-

-RRB-

as

above

Let

Q

be

a

positive

integer

The

Q-truncation

of

D

denoted

Q-Truncation

-LRB-

D

-RRB-

is

the

network

with

dynamics

defined

by

x

Q-Truncation

-LSB-

-LRB-

ss

-LRB-

x

u

-RRB-

-RRB-

-RSB-

where

Q-Truncation

represents

the

operation

of

truncating

after

Q

bits

The

Q-chop

of

D

is

the

network

with

dynamics

defined

by

x

Q-Chop

-LSB-

-LRB-

ss

-LRB-

x

u

-RRB-

-RRB-

-RSB-

Q-Truncation

-LSB-

-LRB-

ssQ

-LRB-

x

u

-RRB-

-RRB-

-RSB-

where

ssQ

is

the

same

polynomial

ss

but

with

coefficients

truncated

after

Q

bits

The

next

observations

insure

that

round-off

errors

due

to

truncation

or

chopping

are

not

too

large

Let

T

IN

IN

be

a

function

from

integers

into

integers

We

say

that

a

generalized

processor

network

D

computes

in

time

T

if

for

every

input

of

size

n

IN

D

completes

its

output

in

no

more

than

T

-LRB-

n

-RRB-

steps

Lemma

Assume

D

computes

in

time

T

with

decision

thresholds

ff

Then

there

is

a

constant

c

such

that

the

function

q

-LRB-

n

-RRB-

cT

-LRB-

n

-RRB-

satisfies

the

following

property

For

each

positive

integer

n

let

Q

q

-LRB-

n

-RRB-

Then

Q-Truncation

-LRB-

D

-RRB-

computes

the

same

function

as

D

on

inputs

of

length

at

most

n

with

decision

thresholds

ff

ff

and

ff

Proof

Let

D

be

a

generalized

processor

network

satisfying

the

above

conditions

and

let

D

Q

truncation

-LRB-

D

-RRB-

with

Q

still

to

be

decided

upon

Let

ffi

be

the

error

due

to

truncating

after

Q

bits

that

is

ffi

for

some

constant

Finally

let

fflt

be

the

largest

accumulated

error

in

all

the

processors

by

time

t

The

following

estimates

are

obtained

using

the

Lipschitz

property

of

f

fflo

ffi

fflt

ffi

t

X

i

o

Ci

ffi

Ct

C

where

C

is

the

Lipschitz

constant

of

f

for

ae

-LRB-

c.f.

Remark

-RRB-

To

bound

the

error

with

the

amount

ff

we

require

fflt

That

is

ffi

-LRB-

C

-RRB-

Ct

C?t

for

some

constant

C

This

requirement

is

met

when

ffi

is

the

truncation

error

corresponding

to

Q

-LRB-

-LRB-

-LRB-

C

-RRB-

-RRB-

T

so

we

can

take

q

-LRB-

n

-RRB-

-LRB-

C

-RRB-

T

-LRB-

n

-RRB-

As

a

corollary

of

Lemma

and

using

an

argument

exactly

as

in

the

proof

of

Lemma

we

conclude

Lemma

Lemma

holds

for

the

Q-chop

network

as

well

Definition

Given

a

vector

function

f

ffi

ss

as

above

we

say

that

f

is

approximable

in

time

Af

-LRB-

n

-RRB-

if

there

is

a

Turing

Machine

M

that

computes

n-Truncation

-LRB-

f

-RRB-

in

time

Af

-LRB-

n

-RRB-

on

each

input

having

total

bit

size

n.

Example

If

f

ffi

ss

is

approximable

and

ss

has

rational

coefficients

then

f

is

approximable

-LRB-

As

ss

is

approximable

at

this

case

-RRB-

Theorem

Let

L

-LRB-

T

-RRB-

be

the

class

of

languages

recognized

by

generalized

processor

networks

in

time

T

for

which

the

function

f

is

approximable

in

time

Af

and

the

function

T

is

computable

in

time

M

-LRB-

n

-RRB-

Then

L

-LRB-

T

-RRB-

is

included

in

the

class

of

languages

recognized

by

Turing

Machines

in

time

O

-LRB-

M

-LRB-

n

-RRB-

T

-LRB-

n

-RRB-

Af

-LRB-

T

-LRB-

n

-RRB-

-RRB-

-RRB-

Proof

Assume

given

a

generalized

processor

network

D

satisfying

the

above

assumptions

A

Turing

Machine

which

approximates

it

can

be

built

as

follows

The

machine

receives

an

input

string

of

length

n

As

a

first

step

it

computes

the

function

T

-LRB-

n

-RRB-

and

it

estimates

the

required

precision

Q

q

-LRB-

n

-RRB-

as

in

the

previous

Lemma

Finally

it

simulates

the

generalized

processor

network

step

by

step

forgetting

all

information

but

the

first

Q

required

bits

This

Turing

Machine

computes

the

required

function

in

the

stated

time

Corollary

Let

D

be

a

network

which

computes

in

polynomial

time

T

and

so

that

f

is

approximable

is

polynomial

time

Then

the

language

recognized

by

D

is

in

P

Definition

Given

a

vector

function

f

ffi

ss

as

above

we

say

that

f

is

non-uniformly

approximable

in

time

Af

-LRB-

n

-RRB-

if

there

is

a

Turing

Machine

M

that

using

a

polynomial

advice

function

-LCB-

c.f.

-LSB-

-RSB-

volume

I

pg

computes

n-Chop

-LRB-

f

-RRB-

in

time

Af

-LRB-

n

-RRB-

on

each

input

having

total

bit

size

n.

Example

A

polynomial

ss

with

general

real

coefficients

is

non-uniformly

computable

For

each

input

of

size

n

the

machine

receives

the

first

O

-LRB-

T

-LRB-

n

-RRB-

-RRB-

bits

of

each

coefficient

as

an

advice

sequence

and

then

computes

the

polynomial

Lemma

Let

L

-LRB-

T

-RRB-

be

the

class

of

languages

recognized

by

generalized

processor

networks

in

time

T

for

which

the

function

f

is

non-uniformly

approximable

in

time

Af

and

the

function

T

is

computable

in

time

M

-LRB-

n

-RRB-

Then

L

-LRB-

T

-RRB-

is

included

in

the

class

of

languages

recognized

by

Turing

Machines

with

advice

sequence

of

length

O

-LRB-

T

-RRB-

in

time

O

-LRB-

M

-LRB-

n

-RRB-

T

-LRB-

n

-RRB-

Af

-LRB-

T

-LRB-

n

-RRB-

-RRB-

-RRB-

Corollary

Let

D

be

a

network

which

computes

in

polynomial

time

T

and

so

that

f

is

nonuniformly

approximable

in

polynomial

time

Then

the

language

recognized

by

D

is

in

P/poly

We

can

summarize

our

conclusion

as

a

general

theorem

as

follows

Theorem

Let

D

be

a

generalized

processor

network

which

computes

via

a

function

f

ffi

ss

where

the

function

is

either

approximable

or

non-uniformly

approximable

in

polynomial

time

Then

there

exists

a

neural

network

ND

which

recognizes

the

same

language

as

D

and

which

does

so

with

at

most

a

polynomial

time

slowdown

Furtheremore

the

weights

utilized

by

ND

are

of

the

same

type

as

the

coefficients

of

the

polynomial

ss

-LRB-

rational

or

real

respectively

-RRB-

Corollary

Adding

flexibility

to

the

neural

network

model

described

in

Equation

-LRB-

-RRB-

does

not

add

power

to

the

model

except

for

a

possible

polynomial

time

speed

up

This

flexibility

includes

ffl

Using

a

high

order

polynomial

ss

rather

than

an

affine

function

ffl

Using

other

functions

rather

than

the

saturation

we

used

earlier

including

the

possibility

of

having

different

functions

in

different

neurons

ffl

Allowing

for

the

output

to

be

soft

binary

rather

than

pure

binary

Note

that

networks

with

high

order

polynomials

have

appeared

especially

in

the

language

recognition

literature

-LRB-

see

e.g.

-LSB-

-RSB-

and

references

there

-RRB-

We

emphasize

the

relationship

between

these

models

Let

be

neural

network

-LRB-

of

any

order

-RRB-

which

recognizes

a

language

L

in

polynomial

time

Then

there

is

a

first

order

network

which

recognizes

the

same

language

L

in

polynomial

time

Remark

The

networks

that

we

consider

are

mildly

robust

to

noise

and

to

implementation

error

in

the

sense

that

small

enough

perturbations

in

weights

or

the

sigmoid

activation

function

do

not

affect

the

computation

as

long

as

soft

binary

outputs

are

considered

Given

any

time

T

there

is

some

fflT

so

that

an

error

of

fflT

would

not

affect

the

computation

up

to

a

time

T

Comments

on

Analog

and

non-Turing

Computation

In

the

recent

very

popular

-LCB-

and

very

controversial

-LCB-

book

-LSB-

-RSB-

Penrose

has

argued

that

the

standard

model

of

computing

is

not

appropriate

for

modeling

true

biological

intelligence

The

author

argues

that

physical

processes

evolving

at

a

quantum

level

may

result

in

computations

which

can

not

be

incorporated

in

Church

's

Thesis

It

is

interesting

to

point

out

that

the

work

that

we

report

here

does

allow

for

such

non-Turing

power

while

keeping

track

of

computational

constraints

-LCB-

and

thus

embedding

a

possible

answer

to

Penrose

's

challenge

in

more

classical

computer

science

Note

that

Parberry

in

-LSB-

-RSB-

also

insists

that

possible

non-Turing

theories

should

take

account

of

such

constraints

though

he

suggests

a

different

approach

namely

the

use

of

probabilistic

computations

within

the

theory

of

circuit

complexity

Finally

we

remark

that

human

cognition

seems

to

be

clearly

based

on

subsymbolic

or

analog

components

and

modes

of

operation

As

pointed

out

by

many

authors

in

particular

in

the

work

of

-LSB-

-RSB-

the

issue

of

understanding

how

macroscopic

symbolic

behavior

arises

from

such

a

substrate

is

one

of

the

most

challenging

ones

in

science

Perhaps

our

work

with

its

implicit

use

of

infinite

precision

for

internal

computations

is

not

at

all

relevant

to

this

understanding

because

neurons

are

often

taken

to

be

low-precision

devices

On

the

other

hand

it

is

also

possible

that

the

precision

issue

should

be

understood

solely

in

terms

of

limitations

on

observers

and

more

generally

interactions

with

the

environment

and

in

that

respect

our

model

is

not

deficient

since

input

and

output

data

are

binary

Acknowledgement

We

wish

to

thank

Richard

Beigel

for

suggesting

that

we

study

the

generalization

of

our

results

to

the

high-order

net

case

treated

in

section

References

-LSB-

-RSB-

Alspector

J.

R.B.

Allen

A

neuromorphic

VLSI

learning

system

in

Advanced

Research

in

VLSI

Proceedings

of

the

Stanford

Conference

-LRB-

P.

Loseleben

ed.

-RRB-

MIT

Press

Cambridge

MA

-LSB-

-RSB-

Atkinson

K.E.

An

Introduction

to

Numerical

Analysis

Wiley

New

York

-LSB-

-RSB-

BalCazar

J.L.

J.

Diaz

J.

Gabarro

Structural

Complexity

Springer-Verlag

Berlin

-LSB-

-RSB-

Blum

L.

M.

Shub

and

S.

Smale

On

a

theory

of

computation

and

complexity

over

the

real

numbers

NP

completeness

recursive

functions

and

universal

machines

Bull

A.M.S.

-LRB-

-RRB-

-LSB-

-RSB-

Chandra

A.K.

L.

Stockmeyer

U.

Vishkin

Constant

depth

reducibility

SIAM

J.

Computing

-LRB-

-RRB-

-LSB-

-RSB-

Eberhardt

S.P.

T.

Daud

D.

A.

Kerns

T.

X.

Brown

and

A.

P.

Thakoor

Competitive

neural

architecture

for

hardware

solution

to

the

assignment

problem

Neural

Networks

-LRB-

-RRB-

-LSB-

-RSB-

Furst

M.

J.B.Saxe

M.

Sipser

Parity

circuits

and

the

polynomial-time

hierarchy

Proc

IEEE

Symp

Foundations

of

Comp

Sci.

-LSB-

-RSB-

Giles

C.L.

C.B.

Miller

D.

Chen

H.H.

Chen

G.Z.

Sun

and

Y.C.

Lee

Grammatical

Inference

Using

Second

Order

Recurrent

Neural

Networks

Proceedings

of

the

International

Joint

Conference

on

Neural

Networks

Seattle

Washington

IEEE

Publication

vol

-LRB-

-RRB-

p.

-LSB-

-RSB-

Hong

J.W.

On

Connectionist

Models

Comm

on

Pure

and

Applied

Mathematics

-LRB-

-RRB-

-LSB-

-RSB-

Karp

R.M.

R.

Lipton

Turing

Machines

that

take

advice

Enseign

Math

-LRB-

-RRB-

-LSB-

-RSB-

Maass

W.

G.

Schnitger

and

E.D.

Sontag

On

the

computational

power

of

sigmoid

versus

Boolean

threshold

circuits

Proc

IEEE

Symp

Foundations

of

Comp

Sci.

-LSB-

-RSB-

MacLennan

B.J.

Continuous

symbol

systems

The

logic

of

connectionism

in

D.S.

Levine

and

M.

Aparicio

IV

-LRB-

eds

-RRB-

Neural

Networks

for

Knowledge

Representation

and

Inference

Lawrence

Erlbaum

Hillsdale

NJ

-LSB-

-RSB-

Muller

D.E.

Complexity

in

electronic

switching

circuits

IRE

Trans

Electronic

Comp

-LRB-

-RRB-

-LSB-

-RSB-

Parberry

I.

Knowledge

understanding

and

computational

complexity

Technical

Report

Center

for

Research

in

Parallel

and

Distributed

Computing

Department

of

Computer

Sciences

University

of

North

Texas

Feb.

-LSB-

-RSB-

Penrose

R.

The

Emperor

's

New

Mind

Oxford

University

Press

Oxford

-LSB-

-RSB-

Savage

J.E.

The

Complexity

of

Computing

New

Tork

Wiley

-LSB-

-RSB-

Siegelmann

H.T.

E.D.

Sontag

On

the

computational

power

of

neural

nets

in

Proc

Fifth

ACM

Workshop

on

Computational

Learning

Theory

Pittsburgh

July

-LSB-

-RSB-

Vergis

A.

K.

Steiglitz

B.

Dickinson

The

complexity

of

analog

computation

in

Math

and

Computers

in

Simulation

-LRB-

-RRB-

-LSB-

-RSB-

Wolpert

D.

A

computationally

universal

field

computer

which

is

purely

linear

Los

Alamos

National

Laboratory

report

analog computation
circuit complexity
dynamical systems
neural networks