What

Size

Neural

Network

Gives

Optimal

Generalization

Convergence

Properties

of

Backpropagation

Steve

C.

Lee

Ah

Chung

flawrence,actg@elec.uq.edu.au,

giles@research.nj.nec.com

NEC

Research

Institute

Independence

Way

Princeton

NJ

Department

of

Electrical

and

Computer

Engineering

University

of

Queensland

St.

Lucia

Australia

Technical

Report

and

Institute

for

Advanced

Computer

Studies

University

of

Maryland

College

Park

MD

Abstract

One

of

the

most

important

aspects

of

any

machine

learning

paradigm

is

how

it

scales

according

to

problem

size

and

complexity

Using

a

task

with

known

optimal

training

error

and

a

pre-speci

ed

maximum

number

of

training

updates

we

investigate

the

convergence

of

the

backpropagation

algorithm

with

respect

to

a

-RRB-

the

complexity

of

the

required

function

approximation

b

-RRB-

the

size

of

the

network

in

relation

to

the

size

required

for

an

optimal

solution

and

c

-RRB-

the

degree

of

noise

in

the

training

data

In

general

for

a

-RRB-

the

solution

found

is

worse

when

the

function

to

be

approximated

is

more

complex

for

b

-RRB-

oversize

networks

can

result

in

lower

training

and

generalization

error

and

for

c

-RRB-

the

use

of

committee

or

ensemble

techniques

can

be

more

bene?cial

as

the

amount

of

noise

in

the

training

data

is

increased

For

the

experiments

we

performed

we

do

not

obtain

the

optimal

solution

in

any

case

We

further

support

the

observation

that

larger

networks

can

produce

better

training

and

generalization

error

using

a

face

recognition

example

where

a

network

with

many

more

parameters

than

training

points

generalizes

better

than

smaller

networks

yAlso

with

the

Institute

for

Advanced

Computer

Studies

University

of

Maryland

College

Park

MD

z

Introduction

Statements

regarding

the

training

and

generalization

error

of

MLPs

similar

to

the

following

occur

often

in

the

neural

network

literature

and

community

BP

is

actually

a

gradient

method

and

therefore

there

is

no

guarantee

at

all

that

the

absolute

minimum

can

be

reached

In

spite

of

this

theoretical

remark

researchers

involved

with

BP

applications

know

that

this

is

not

a

very

serious

problem

BP

often

leads

to

a

global

minimum

or

at

least

makes

it

possible

to

meet

practical

stopping

criteria

We

have

found

local

minima

to

be

very

rare

and

that

the

system

learns

in

a

reasonable

period

of

time

Backpropagation

works

well

by

avoiding

non-optimal

solutions

We

should

not

use

a

network

with

more

parameters

than

the

number

of

data

points

available

Statements

to

say

that

while

local

minima

are

expected

they

nevertheless

either

do

not

affect

the

quality

of

the

solution

greatly

or

they

occur

so

infrequently

that

the

effect

can

be

ignored

in

practice

Statement

expresses

the

intuition

that

the

degrees

of

freedom

in

the

model

should

be

less

than

the

total

number

of

data

points

available

for

training

In

this

paper

we

show

that

a

solution

near

the

optimal

solution

is

often

not

obtained

We

investigate

the

relative

quality

of

the

solution

obtained

as

a

function

of

the

following

variables

a

-RRB-

the

complexity

of

the

required

function

approximation

b

-RRB-

the

size

of

the

network

in

relation

to

the

size

required

for

an

optimal

solution

and

c

-RRB-

the

degree

of

noise

present

in

the

data

We

show

that

a

-RRB-

the

solution

found

is

worse

when

the

function

to

be

approximated

is

more

complex

b

-RRB-

oversize

networks

can

result

in

lower

training

and

generalization

error

and

c

-RRB-

the

use

of

committee

or

ensemble

techniques

can

be

more

bene?cial

as

the

amount

of

noise

in

the

training

data

is

increased

We

further

support

the

observation

that

larger

networks

can

produce

better

training

and

generalization

error

using

a

face

recognition

example

where

a

network

with

times

more

parameters

than

training

points

generalizes

better

than

smaller

networks

We

do

not

use

any

techniques

to

control

generalization

in

order

to

illustrate

this

case

Local

Minima

It

is

important

to

de?ne

what

we

mean

by

a

local

minimum

or

a

local

optimum

It

has

been

shown

that

the

error

surface

of

a

backpropagation

network

with

one

hidden

layer

and

t

hidden

units

has

no

local

minima

if

the

network

is

trained

with

an

arbitrary

set

containing

t

different

-LRB-

Yu

-RRB-

We

note

that

in

practice

it

can

be

dif?cult

to

distinguish

a

true

local

minimum

from

other

dif?cult

error

surface

features

such

as

ravines

or

plateaus

-LRB-

Baldi

and

Hornik

-RRB-

in

the

standard

backpropagation

algorithm

For

the

purposes

of

this

paper

we

de?ne

the

criterion

of

interest

as

the

best

solution

found

in

a

given

practical

time

limit

large

t

the

method

may

be

impractical

due

to

the

size

of

the

network

required

to

ensure

that

there

are

no

local

minima

Prior

Work

The

error

surface

of

very

small

networks

has

been

characterized

previously

e.g.

for

an

XOR

network

-LRB-

Hamey

-RRB-

However

practical

networks

often

contain

hundreds

or

thousands

of

and

in

general

theoretical

and

empirical

results

on

small

networks

do

not

scale

up

to

large

networks

One

reason

may

be

attributed

to

the

interference

effect

in

the

training

process

Consider

the

familiar

backpropagation

training

algorithm

if

the

hidden

layer

neurons

are

not

all

in

saturation

the

gradients

evaluated

at

the

hidden

layer

neurons

are

coupled

-LRB-

the

update

of

each

parameter

generally

affects

many

other

parameters

-RRB-

For

a

network

with

more

hidden

layer

neurons

this

interference

effect

is

expected

to

be

more

pronounced

Caruana

presented

a

tutorial

at

NIPS

-LRB-

Caruana

-RRB-

with

generalization

results

on

a

variety

of

problems

as

the

size

of

the

networks

was

varied

from

too

small

to

too

large

Too

small

and

too

large

are

related

to

the

number

of

parameters

in

the

model

-LRB-

without

consideration

of

the

distribution

of

the

data

the

error

surface

etc.

-RRB-

Caruana

reported

that

large

networks

rarely

do

worse

than

small

networks

on

the

problems

he

investigated

The

results

in

this

paper

partially

correlate

with

that

observation

Caruana

suggests

that

backprop

ignores

excess

parameters

Crane

et

al.

-LRB-

Crane

et

al.

-RRB-

used

real-valued

data

generated

by

a

random

target

network

and

attempted

training

new

networks

on

the

data

in

order

to

approximate

the

number

of

minima

on

the

error

surface

under

varying

conditions

The

use

of

random

target

networks

in

this

fashion

has

been

referred

to

as

the

student

teacher

problem

-LRB-

Saad

and

Solla

-RRB-

Motivated

by

this

work

we

use

a

very

similar

technique

to

evaluate

the

quality

of

the

local

minima

which

are

found

using

backpropagation

as

a

function

of

various

parameters

Saad

and

Solla

-LRB-

Saad

and

Solla

-RRB-

have

used

the

student

teacher

problem

to

analyze

the

effect

of

noise

on

on-line

learning

For

the

case

of

training

examples

corrupted

with

additive

output

noise

-LRB-

they

also

analyze

model

noise

-RRB-

Saad

and

Solla

have

shown

that

small

noise

levels

may

shorten

the

symmetric

phase

of

learning

while

larger

values

may

lengthen

the

phase

Generalization

error

increases

as

the

noise

level

is

increased

For

the

asymptotic

case

training

with

a

xed

learning

rate

results

in

a

non-vanishing

asymptotic

generalization

error

They

show

that

learning

rate

decay

schemes

can

remove

the

effects

of

additive

output

noise

asymptotically

M?uller

et

al.

-LRB-

M?uller

et

al.

-RRB-

also

used

randomly

generated

teacher

networks

in

order

to

create

training

examples

for

student

networks

They

perform

a

detailed

study

of

generalization

as

a

function

of

the

number

of

training

samples

for

classi?cation

For

networks

with

up

to

weights

they

demonstrate

strong

over?tting

for

a

small

number

of

training

examples

a

region

where

the

generalization

error

scales

according

to

where

ntraining

is

the

number

of

training

examples

and

asymptotic

scaling

according

to

nweights

where

nweights

is

the

number

of

weights

in

the

network

There

are

several

theories

for

determining

the

optimal

network

size

e.g.

the

NIC

-LRB-

Network

Information

Criterion

-RRB-

-LRB-

Amari

-RRB-

which

is

a

generalization

of

the

AIC

-LRB-

Akaike

Information

Criterion

-RRB-

-LRB-

Akaike

Akaike

-RRB-

widely

used

in

statistical

inference

the

generalized

nal

prediction

error

-LRB-

GPE

-RRB-

as

proposed

by

Moody

-LRB-

Moody

-RRB-

and

the

Vapnik-Chervonenkis

-LRB-

VC

-RRB-

dimension

-LRB-

Maass

Abu

with

up

to

million

weights

have

been

used

for

speech

phoneme

recognition

-LRB-

Bourlard

and

Morgan

-RRB-

respect

to

the

results

reported

here

over?tting

behaviour

for

classi?cation

tasks

is

expected

to

be

different

due

to

the

use

of

training

patterns

with

asymptotic

targets

nal

prediction

error

-LRB-

FPE

-RRB-

is

an

alternative

method

for

determining

the

order

of

a

dynamical

process

originally

proposed

by

Akaike

-LRB-

Akaike

-RRB-

and

generalized

to

the

neural

network

setting

by

Moody

-LRB-

Moody

-RRB-

Mostafa

Bartlett

-RRB-

which

is

a

measure

of

the

expressive

power

of

a

network

NIC

relies

on

a

single

well-de

ned

minimum

to

the

tting

function

and

can

be

unreliable

when

there

are

several

local

minima

-LRB-

Ripley

-RRB-

There

is

very

little

published

computational

experience

of

the

NIC

or

the

GPE

Their

evaluation

is

prohibitively

expensive

for

large

networks

VC

bounds

have

been

calculated

for

various

network

types

-LRB-

Cohn

and

Tesauro

-RRB-

Early

VC-dimension

work

handles

only

the

case

of

discrete

outputs

For

the

case

of

real

valued

outputs

a

more

general

notion

of

a

dimension

is

required

Such

a

pseudo-dimension

can

be

de?ned

by

considering

a

loss

function

which

measures

the

deviation

of

predictions

from

the

target

values

-LRB-

Maass

-RRB-

VC

bounds

are

likely

to

be

too

conservative

because

they

provide

generalization

guarantees

simultaneously

for

any

probability

distribution

and

any

training

algorithm

The

computation

of

VC

bounds

for

practical

networks

is

dif?cult

Apart

from

small

examples

we

are

unaware

of

any

systematic

procedures

for

the

evaluation

of

VC

bounds

for

typical

practical

networks

Other

work

addressing

local

minima

or

the

number

of

samples

required

with

respect

to

generalization

include

-LRB-

Baum

and

Haussler

Sartori

and

Antsaklis

McInerny

et

al.

Yu

Gori

and

Tesi

-RRB-

These

approaches

are

limited

due

to

the

assumptions

they

make

e.g.

typical

limitations

include

applicability

only

to

linearly

separable

problems

consideration

only

of

true

local

minima

as

opposed

to

regions

where

gradient

descent

becomes

stuck

and

no

consideration

of

limits

on

training

time

Arti?cial

Task

To

investigate

empirical

performance

we

have

chosen

an

arti?cial

task

so

that

we

a

-RRB-

know

the

optimal

solution

and

b

-RRB-

can

carefully

control

various

parameters

We

aim

to

make

the

task

as

realistic

as

possible

by

using

real-valued

inputs

and

outputs

The

task

is

as

follows

and

is

very

similar

to

the

procedure

used

in

-LRB-

Crane

et

al.

-RRB-

An

MLP

with

mi

input

nodes

mh

hidden

nodes

and

mo

output

nodes

-LRB-

denoted

by

mi

mh

mo

and

later

referred

to

as

the

data

generating

network

-RRB-

is

initialized

with

random

weights

uniformly

selected

within

a

speci?ed

range

i.e.

wi

in

the

range

K

to

K

where

wi

are

the

weights

of

the

network

except

the

biases

and

K

is

a

constant

K

is

for

the

results

reported

in

this

paper

except

when

otherwise

speci?ed

The

bias

weights

are

initialized

to

small

random

values

in

the

range

-LRB-

-RRB-

As

K

is

increased

the

complexity

of

the

function

mapping

is

increased

as

will

be

discussed

in

more

detail

in

section

ntraining

data

points

are

created

by

selecting

random

inputs

with

zero

mean

and

unit

variance

and

propagating

them

through

the

network

to

nd

the

corresponding

outputs

This

dataset

S

forms

the

training

data

for

subsequent

simulations

The

procedure

is

repeated

to

create

a

test

dataset

with

ntest

points

ntest

is

for

all

simulations

reported

in

this

paper

The

choice

of

zero

mean

and

unit

variance

inputs

is

not

too

unrealistic

because

the

inputs

to

an

MLP

are

often

normalised

to

have

zero

mean

and

unit

variance

-LRB-

the

difference

is

that

the

distribution

may

not

be

normal

-RRB-

-LRB-

Le

Cun

-RRB-

brie?y

this

is

the

largest

set

of

examples

that

can

be

shattered

by

the

network

where

a

set

of

x

examples

is

shattered

by

the

network

if

for

each

of

the

possible

ways

of

dividing

the

x

samples

into

disjoint

sets

and

there

exists

a

function

computable

by

the

network

such

that

the

output

is

for

members

of

and

the

output

is

for

members

of

-LRB-

for

a

binary

classi?cation

problem

-RRB-

The

training

data

set

S

is

used

to

train

new

MLPs

known

subsequently

as

the

trained

networks

with

the

following

architecture

mi

mo.

For

certain

tests

is

varied

from

mh

to

M

where

M

mh

The

initial

weights

of

these

new

networks

are

set

using

standard

procedures

-LRB-

i.e.

they

are

not

equal

to

the

weights

in

the

network

used

to

create

the

dataset

-RRB-

They

are

initialized

on

a

node

by

node

basis

as

uniformly

distributed

random

numbers

in

the

range

-LRB-

Fi

Fi

-RRB-

where

Fi

is

the

fan-in

of

neuron

i

as

shown

in

-LRB-

Haykin

-RRB-

Theoretically

if

mh

then

the

optimal

training

set

error

is

zero

Figure

shows

the

process

graphically

Figure

The

process

of

creating

the

data

sets

Methodology

for

Exploring

Convergence

We

are

going

to

use

the

previous

arti?cial

task

to

explore

the

convergence

of

the

networks

while

varying

certain

parameters

in

a

controlled

manner

We

examine

both

the

training

and

the

generalization

performance

We

use

the

topology

as

our

baseline

topology

where

and

were

chosen

to

represent

a

typical

network

where

the

number

of

inputs

is

greater

than

the

number

of

hidden

nodes

and

the

speci?c

values

were

chosen

such

that

the

total

training

time

of

the

simulations

was

reasonable

We

use

the

following

methodology

We

vary

the

following

parameters

of

the

simulations

one

at

a

time

a

-RRB-

the

maximum

value

used

for

setting

the

weights

in

the

generating

network

-LRB-

K

-RRB-

b

-RRB-

the

size

of

the

trained

networks

-LRB-

c

-RRB-

the

size

of

the

training

dataset

-LRB-

ntraining

and

d

-RRB-

the

amount

of

zero

mean

Gaussian

noise

added

to

the

training

data

-LRB-

from

zero

to

a

standard

deviation

of

of

the

standard

deviation

of

the

input

data

-RRB-

Each

con?guration

of

the

MLP

is

tested

with

multiple

simulations

-LRB-

-RRB-

each

with

a

different

starting

condition

Stopping

criterion

We

speci?cally

do

not

consider

a

stopping

criterion

other

than

a

maximum

number

of

training

updates

-LRB-

for

all

experiments

conducted

in

this

paper

-RRB-

All

networks

are

trained

for

an

identical

duration

-LRB-

the

number

of

epochs

is

adjusted

depending

on

the

number

of

training

examples

such

that

the

total

number

of

stochastic

updates

-RRB-

We

do

not

consider

controlling

generalization

as

we

wish

to

demonstrate

this

We

expect

that

over?tting

could

occur

We

used

the

standard

MLP

ylk

f

PNl

i

wlkiyl

i

where

ylk

is

the

output

of

neuron

k

in

layer

l

Nl

is

the

number

of

neurons

in

layer

l

wlki

is

the

weight

connecting

neuron

k

in

layer

l

to

neuron

i

in

layer

l

-LRB-

bias

-RRB-

and

f

is

the

hyperbolic

tangent

function

The

number

of

weights

in

each

network

is

thus

-LRB-

mi

-RRB-

mh

-LRB-

mh

-RRB-

mo.

We

used

standard

backpropagation

with

stochastic

update

-LRB-

update

after

each

training

point

-RRB-

We

also

investigated

batch

update

and

found

that

results

were

very

poor

convergence

was

very

poor

even

when

training

times

were

extended

by

an

order

of

magnitude

The

quadratic

cost

function

was

used

E

PNi

PNi

Pmo

k

-LRB-

dki

yki

-RRB-

where

dki

is

the

desired

value

of

the

k

th

output

neuron

for

the

i

th

training

sample

from

the

training

data

set

S

and

yki

is

the

value

of

k

th

output

neuron

of

the

MLP

in

response

to

the

i

th

training

sample

The

learning

rate

was

Simulation

Results

We

present

the

results

of

varying

the

network

size

the

training

set

size

the

function

complexity

and

the

amount

of

noise

added

to

the

training

data

Network

Size

In

this

section

we

investigate

the

training

and

generalization

behavior

of

the

networks

with

the

generating

network

size

xed

but

the

trained

network

size

increasing

Simulation

results

are

shown

in

gure

For

all

cases

the

data

was

created

with

a

generating

network

architecture

and

random

weight

maximum

value

K

All

trained

networks

have

the

following

architecture

The

graphs

show

the

training

and

test

set

errors

as

the

number

of

hidden

units

is

increased

from

-LRB-

left

-RRB-

to

-LRB-

right

-RRB-

ntraining

and

ntest

The

number

of

parameters

in

the

networks

is

shown

in

table

There

are

more

parameters

than

the

number

of

data

points

and

as

such

we

might

expect

Theoretically

are

many

ways

of

controlling

generalization

e.g.

a

-RRB-

early

stopping

b

-RRB-

weight

decay

or

weight

elimination

and

c

-RRB-

pruning

e.g.

OBD

-LRB-

optimal

brain

damage

-RRB-

-LRB-

Le

Cun

et

al.

-RRB-

and

OBS

-LRB-

optimal

brain

surgeon

-RRB-

-LRB-

Hassibi

and

Stork

-RRB-

trained

networks

smaller

than

the

data

generating

network

the

training

and

generalization

error

follow

an

expected

increasing

trend

as

the

network

size

is

reduced

the

optimal

training

set

error

for

all

networks

tested

is

zero

as

mh

However

we

were

unable

to

train

any

network

to

this

optimal

error

for

this

problem

As

the

number

of

hidden

units

is

increased

we

observe

that

a

better

solution

is

found

Figure

shows

the

results

for

data

points

which

is

less

than

the

number

of

parameters

in

all

of

the

models

Figure

indicates

that

better

generalization

performance

is

obtained

for

all

of

the

networks

which

are

larger

than

the

optimal

size

The

best

generalization

error

corresponds

to

the

networks

with

hidden

nodes

We

note

that

this

trend

varies

according

to

the

generating

network

size

-LRB-

number

of

inputs

hidden

nodes

and

outputs

-RRB-

and

the

nature

of

the

required

approximation

e.g.

over?tting

can

be

seen

for

if

the

task

is

made

simple

enough

or

in

other

cases

the

advantage

of

larger

networks

can

be

even

greater

Train

NMSE

Number

Of

Hidden

Nodes

Test

NMSE

Number

Of

Hidden

Nodes

Figure

The

error

for

networks

with

a

topology

The

graph

on

the

top

is

the

training

error

The

graph

on

the

bottom

is

the

test

error

The

abscissa

corresponds

to

the

number

of

hidden

nodes

Each

result

is

averaged

over

ve

simulations

plus

and

minus

one

standard

deviation

error

bars

are

plotted

at

each

point

Number

of

hidden

nodes

Number

of

parameters

221 441 661 881

Table

The

number

of

parameters

in

the

networks

displayed

in

gure

Figure

shows

the

results

for

and

data

points

We

observe

similar

trends

however

the

error

in

general

is

lower

as

the

number

of

points

is

increased

In

each

of

these

cases

the

best

generalization

error

was

obtained

with

networks

containing

hidden

nodes

It

is

of

interest

to

observe

the

effect

of

noise

on

this

problem

Figure

shows

the

results

when

Gaussian

noise

is

added

to

the

input

data

with

a

standard

deviation

equal

to

of

the

standard

deviation

of

the

input

data

We

observe

a

similar

trend

to

gure

The

best

generalization

error

is

obtained

for

networks

containing

hidden

nodes

Note

that

we

have

deliberately

provided

no

control

of

the

generalization

capability

of

the

networks

There

are

many

solutions

which

t

the

training

data

well

that

will

not

generalize

well

Yet

we

still

nd

contrary

to

popular

beliefs

that

oversize

networks

can

provide

better

generalization

We

note

that

successive

pruning

Train

NMSE

Number

Of

Hidden

Nodes

training

points

Test

NMSE

Number

Of

Hidden

Nodes

training

points

Train

NMSE

Number

Of

Hidden

Nodes

training

points

Test

NMSE

Number

Of

Hidden

Nodes

training

points

Figure

The

error

for

the

trained

networks

as

shown

in

gure

From

top

to

bottom

training

points

training

error

training

points

test

error

training

points

training

error

training

points

test

error

and

retraining

of

a

larger

network

may

arrive

at

a

network

with

similar

size

to

the

smaller

networks

here

but

with

improved

training

and

generalization

error

-LRB-

Hassibi

and

Stork

-RRB-

Degrees

of

Freedom

Rules

based

on

the

degrees

of

freedom

in

the

model

have

been

proposed

for

selecting

the

topology

of

an

MLP

e.g.

The

number

of

parameters

in

the

network

should

be

-LRB-

signi?cantly

-RRB-

less

than

the

number

of

examples

or

Each

parameter

in

an

MLP

can

comfortably

store

bits

of

information

A

network

with

more

than

this

will

tend

to

memorize

the

data

-LRB-

known

as

the

CMU

rule

of

thumb

-RRB-

Train

NMSE

Number

Of

Hidden

Nodes

Test

NMSE

Number

Of

Hidden

Nodes

Figure

The

error

for

the

trained

networks

as

shown

in

gure

with

noise

added

to

the

input

data

The

noise

was

Gaussian

with

a

standard

deviation

equal

to

of

the

standard

deviation

of

the

original

data

Top

to

bottom

training

and

test

errors

These

rules

aim

to

prevent

over?tting

but

they

are

unreliable

as

the

optimal

value

is

likely

to

depend

on

the

quality

of

the

solution

found

the

distribution

of

the

data

the

amount

of

noise

the

nature

of

the

function

being

approximated

etc.

The

speci?c

rules

are

not

commonly

believed

to

be

accurate

However

the

stipulation

that

the

number

of

parameters

must

be

less

than

the

number

of

examples

is

typically

believed

to

be

true

for

common

datasets

We

have

shown

that

this

is

not

always

the

case

in

the

previous

section

We

present

some

results

on

real

data

in

order

to

highlight

our

point

Figure

shows

the

results

of

training

an

MLP

to

classify

people

from

images

of

their

The

training

set

contains

images

per

person

for

a

total

of

training

The

test

set

contained

a

different

set

of

images

per

person

A

small

window

was

stepped

over

the

images

and

the

image

samples

at

each

point

were

quantized

using

a

two

dimensional

self-organizing

map

-LRB-

Kohonen

-RRB-

The

outputs

of

the

self-organizing

map

for

each

image

sample

were

used

as

the

inputs

to

the

MLP

A

subset

of

the

images

is

shown

in

gure

In

each

case

the

networks

were

trained

for

updates

The

networks

used

contain

many

more

parameters

than

the

number

of

training

points

as

shown

in

gure

yet

the

best

training

error

and

the

best

generalization

error

corresponds

to

the

largest

model

Note

that

we

have

not

controlled

the

generalization

in

any

way

When

simulated

on

serial

machines

larger

networks

require

additional

computational

resources

Hence

the

case

where

data

points

lie

in

clusters

e.g.

the

extreme

case

where

some

data

points

are

identical

do

not

propose

this

as

an

intelligent

face

recognition

technique

have

used

the

ORL

database

which

contains

a

set

of

faces

taken

between

April

and

April

at

the

Olivetti

Research

Laboratory

in

Cambridge

and

is

available

from

There

are

different

images

of

distinct

subjects

in

the

database

There

are

variations

in

facial

expression

and

facial

details

All

the

images

are

taken

against

a

dark

homogeneous

background

with

the

subjects

in

an

up-right

frontal

position

with

tolerance

for

some

tilting

and

rotation

of

up

to

about

degrees

There

is

some

variation

in

scale

of

up

to

about

The

images

are

greyscale

-LRB-

levels

-RRB-

with

a

resolution

of

it

is

of

interest

to

compare

what

happens

when

the

smaller

networks

are

trained

for

longer

than

the

larger

networks

Figure

presents

the

results

of

training

all

of

the

networks

twice

as

long

We

observe

that

the

results

do

not

change

signi?cantly

-LRB-

for

example

compare

the

error

for

the

networks

with

hidden

nodes

and

the

extended

training

networks

with

hidden

nodes

-RRB-

Figure

A

subset

of

the

face

images

used

in

the

experiments

Test

Error

Number

Of

Hidden

Nodes

Number

of

hidden

nodes

Number

of

weights

Figure

Face

recognition

experiments

showing

that

the

optimal

number

of

weights

in

a

network

can

be

much

larger

than

the

number

of

data

points

The

smallest

model

with

six

hidden

has

times

more

parameters

than

training

points

-LRB-

parameters

-RRB-

The

largest

model

with

hidden

nodes

has

times

more

parameters

than

training

points

-LRB-

parameters

-RRB-

The

test

error

is

given

as

the

percentage

of

examples

incorrectly

classi?ed

Test

Error

Number

Of

Hidden

Nodes

Figure

As

per

gure

with

the

networks

trained

for

twice

as

long

Function

Complexity

In

this

section

we

consider

the

quality

of

the

solution

found

by

backpropagation

as

the

maximum

random

weight

in

the

generating

network

K

is

increased

Figure

shows

the

results

for

target

and

generating

networks

with

topology

ntraining

and

ntest

It

is

dif?cult

to

visualize

the

function

we

are

trying

to

approximate

as

we

increase

K

A

simple

method

which

gives

us

an

indication

is

plotted

in

gure

and

is

created

as

follows

for

each

output

o

for

each

input

i

set

all

inputs

i

equal

to

plot

output

o

as

input

i

is

varied

from

to

repeat

times

set

all

inputs

i

equal

to

Gaussian

random

values

with

mean

and

variance

plot

output

o

as

input

i

is

varied

from

to

K

K

Figure

Plots

indicating

the

complexity

of

the

function

we

are

asking

the

networks

to

approximate

Above

K

below

K

The

networks

used

for

creating

these

plots

had

the

topology

The

rows

correspond

to

the

rst

ve

inputs

of

these

networks

The

rst

column

corresponds

to

the

case

where

all

other

inputs

are

set

to

zero

and

the

remaining

columns

correspond

to

the

cases

where

the

other

inputs

are

set

to

random

values

as

per

the

pseudo-code

The

abscissa

of

each

individual

plot

corresponds

to

the

value

of

the

input

for

that

row

and

ranges

from

to

The

ordinate

of

each

individual

plot

corresponds

to

the

network

output

From

gure

we

can

observe

that

the

function

complexity

increases

when

K

is

increased

from

to

Hence

K

may

be

considered

as

a

parameter

controlling

the

function

complexity

although

we

are

not

using

a

precise

de?nition

of

complexity

Again

the

optimal

training

set

error

for

all

networks

is

zero

because

mh

As

K

increases

corresponding

to

the

function

which

we

are

asking

the

network

to

approximate

becoming

more

complex

we

observe

that

the

solution

found

becomes

signi?cantly

worse

Note

that

we

may

expect

generalization

to

be

worse

as

K

increases

however

we

are

mainly

interested

in

the

training

error

when

compared

to

the

optimal

error

of

These

results

highlight

a

point

regarding

the

application

of

neural

network

models

the

nature

of

the

function

we

are

asking

the

network

to

approximate

is

often

not

considered

in

detail

Perhaps

consideration

of

the

implicit

bias

towards

smooth

models

can

be

of

help

and

preprocessing

efforts

could

be

directed

towards

formulating

the

required

approximation

to

better

suit

the

neural

network

Additionally

we

expect

that

if

the

required

approximation

is

smoother

then

the

weights

of

the

network

are

less

likely

to

be

driven

towards

in?nity

nodes

are

less

likely

to

become

saturated

and

generalization

performance

may

be

expected

to

improve

Train

NMSE

Maximum

Random

Weight

Test

NMSE

Maximum

Random

Weight

Figure

Error

for

networks

with

the

topology

as

the

maximum

random

weight

for

the

generating

network

is

increased

The

graph

on

the

top

is

the

training

error

The

graph

on

the

bottom

is

the

test

error

The

abscissa

corresponds

to

the

value

of

K.

Each

result

is

averaged

over

ve

simulations

plus

and

minus

one

standard

deviation

error

bars

are

plotted

at

each

point

Noise

Committees

or

an

ensemble

of

networks

are

known

to

be

able

to

improve

generalization

performance

-LRB-

Jacobs

Drucker

et

al.

-RRB-

In

this

section

we

investigate

the

effect

of

using

committees

as

the

amount

of

noise

added

to

the

training

data

increases

We

use

a

simple

weighted

ensemble

of

networks

Consider

the

bias/variance

dilemma

as

in

-LRB-

Geman

et

al.

-RRB-

where

the

MSE

may

be

decomposed

into

bias

and

variance

components

MSEbias

-LRB-

ED

-LSB-

f

-LRB-

x

-RRB-

-RSB-

E

-LSB-

yjx

-RSB-

-RRB-

-LRB-

-RRB-

MSEvariance

ED

-LRB-

f

-LRB-

x

-RRB-

ED

-LSB-

f

-LRB-

x

-RRB-

-RSB-

-RRB-

-LRB-

-RRB-

where

ED

represents

the

expectation

with

respect

to

a

training

set

D

and

f

-LRB-

x

-RRB-

is

the

approximating

function

With

nite

training

data

reducing

the

bias

generally

increases

the

variance

and

vice

versa

For

a

multi-layer

perceptron

there

is

another

variance

term

due

to

convergence

to

local

minima

which

can

be

reduced

using

ensembles

and

the

effect

of

this

reduction

is

greater

if

the

individual

networks

have

larger

variance

-LRB-

see

-LRB-

Naftaly

et

al.

-RRB-

-RRB-

Increasing

noise

levels

and

the

resulting

poorer

convergence

may

induce

this

condition

Hence

we

expect

that

the

ensemble

technique

may

be

more

bene?cial

as

the

noise

level

is

increased

Figure

shows

the

results

of

using

to

committee

networks

as

the

standard

deviation

of

zero

mean

Gaussian

noise

added

to

the

training

data

is

varied

from

to

of

the

standard

deviation

of

the

original

data

We

observe

that

the

use

of

more

networks

in

the

ensemble

does

appear

to

be

more

successful

as

the

noise

is

increased

Test

NMSE

Number

Of

Committee

Networks

no

noise

Test

NMSE

Number

Of

Committee

Networks

noise

Test

NMSE

Number

Of

Committee

Networks

noise

Figure

Test

error

as

the

number

of

committee

networks

is

increased

from

to

in

each

plot

From

top

to

bottom

no

noise

and

Gaussian

noise

with

a

standard

deviation

of

and

of

the

standard

deviation

of

the

original

data

respectively

Discussion

Using

an

arti?cial

task

where

we

knew

the

optimal

training

error

and

a

sample

real

world

problem

along

with

a

pre-speci

ed

maximum

number

of

training

updates

we

showed

that

The

solution

found

by

gradient

descent

is

often

signi?cantly

worse

than

the

global

minimum

In

certain

cases

the

best

generalization

can

occur

when

the

number

of

parameters

in

the

network

is

greater

than

the

number

of

data

patterns

There

is

a

class

of

-LRB-

more

complex

-RRB-

problems

in

which

over?tting

may

not

a

problem

with

reasonable

training

times

The

solution

found

by

gradient

descent

is

worse

as

the

function

to

be

approximated

becomes

more

complex

The

use

of

ensemble

techniques

can

be

more

bene?cial

as

the

degree

of

noise

in

the

training

data

increases

Given

the

set

of

function

that

a

particular

MLP

can

approximate

and

a

pre-speci

ed

maximum

number

of

training

updates

certain

functions

are

harder

to

approximate

using

backpropagation

than

others

These

functions

tend

to

be

those

where

an

increasing

number

of

nodes

are

in

the

saturation

region

for

a

percentage

of

the

inputs

In

the

following

sections

we

discuss

related

theory

which

supports

the

results

Rate

of

Convergence

The

rate

of

approximation

of

a

function

in

relation

to

the

function

's

smoothness

properties

has

been

studied

in

classical

approximation

theory

where

smoothness

is

usually

measured

by

the

number

of

existing

derivatives

Typical

results

for

the

asymptotic

rate

of

convergence

have

the

form

-LRB-

Vapnik

-RRB-

rn

n

smi

-LRB-

-RRB-

where

rn

is

the

rate

of

convergence

n

is

the

number

of

parameters

s

is

the

number

of

existing

derivatives

of

the

function

to

approximate

and

mi

is

the

dimensionality

of

the

input

space

-LRB-

Lorentz

-RRB-

Lower

rn

corresponds

to

faster

convergence

and

rn

pmi

is

considered

to

be

a

fast

rate

of

convergence

-LRB-

Vapnik

-RRB-

Mhaskar

-LRB-

Mhaskar

-RRB-

showed

that

the

rate

of

approximation

in

MLP

models

is

also

of

O

-LRB-

n

sN

-RRB-

The

results

indicate

that

in

order

to

guarantee

a

high

asymptotic

rate

of

convergence

the

function

to

approximate

must

be

smoother

as

the

dimensionality

increases

Hence

given

a

nite

training

time

the

faster

convergence

for

smoother

functions

induces

a

bias

towards

smoother

solutions

M?uller

et

al.

-LRB-

M?uller

et

al.

-RRB-

examine

the

case

where

the

number

of

weights

in

the

network

is

larger

than

the

number

of

training

examples

for

a

classi?cation

task

The

weights

are

theoretically

expected

to

diverge

however

this

is

not

observed

in

typical

simulations

due

to

the

atness

of

the

error

surface

and

numerical

effects

of

nite

precision

which

induce

an

implicit

bias

towards

smoother

functions

Occam

's

Razor

The

results

showing

that

larger

than

optimal

size

networks

can

generalize

better

are

not

in

contradiction

with

Occam

's

razor

Occam

's

razor

which

advocates

the

simpler

out

of

a

number

of

possible

solutions

is

not

applicable

to

the

situation

where

each

solution

is

of

a

different

quality

i.e.

while

larger

networks

can

provide

improved

generalization

performance

these

networks

are

better

models

of

the

training

data

A

conclusion

is

that

the

nature

of

backpropagation

results

in

the

underutilisation

of

network

resources

in

certain

cases

Learning

Theory

The

results

are

also

not

in

contradiction

with

statistical

learning

theory

Vapnik

-LRB-

Vapnik

-RRB-

states

that

machines

with

a

small

VC

dimension

are

required

to

avoid

over?tting

However

he

also

states

that

it

is

dif?cult

to

approximate

the

training

data

Vapnik

suggests

that

the

use

of

a

priori

knowledge

is

required

for

a

small

training

error

and

small

generalisation

error

i.e.

for

a

given

problem

in

MLP

approximation

we

need

to

nd

the

appropriate

network

size

in

order

to

minimize

the

tradeoff

between

over?tting

and

poor

approximation

For

the

case

of

linear

output

neurons

Barron

-LRB-

Barron

Barron

-RRB-

has

derived

the

following

bound

on

the

total

risk

for

an

MLP

estimator

O

Cf

mh

O

mhmi

ntraining

log

ntraining

-LRB-

-RRB-

where

Cf

is

the

rst

absolute

moment

of

the

Fourier

magnitude

distribution

of

the

target

function

f

and

is

a

measure

of

the

complexity

of

f

Again

we

observe

a

tradeoff

between

the

accuracy

of

the

best

approximation

-LRB-

which

requires

larger

mh

-RRB-

and

the

avoidance

of

over?tting

-LRB-

which

requires

a

smaller

mh

ntraining

ratio

-RRB-

However

this

does

not

take

into

account

limited

training

time

and

different

rates

of

convergence

for

different

f

The

Curse

of

Dimensionality

Consider

xi

Rn

The

regression

f

-LRB-

x

-RRB-

is

a

hypersurface

in

Rn

If

f

-LRB-

x

-RRB-

is

arbitrarily

complex

and

unknown

then

dense

samples

are

required

to

approximate

the

function

accurately

However

it

is

hard

to

obtain

dense

samples

in

high

dimensions

This

is

the

curse

of

dimensionality

-LRB-

Friedman

-RRB-

The

relationship

between

the

sampling

density

and

the

number

of

points

required

is

ss

N

-LRB-

Friedman

-RRB-

where

n

is

the

dimensionality

of

the

input

space

and

N

is

the

number

of

points

Thus

if

is

the

number

of

points

for

to

local

minima

plateaus

etc.

a

given

sampling

density

in

dimension

then

in

order

to

keep

the

same

density

as

the

dimensionality

is

increased

the

number

of

points

must

increase

according

to

Kolmogorov

's

solution

to

Hilbert

's

problem

shows

that

any

continuous

function

of

n

dimensions

can

be

completely

characterized

by

a

continuous

function

What

does

this

say

The

problem

is

not

so

much

the

dimensionality

but

the

complexity

of

the

function

-LRB-

high

dimensional

functions

typically

have

the

potential

to

be

more

complex

-RRB-

i.e.

the

less

data

points

the

more

dimensions

the

more

complex

the

required

approximation

the

more

ill-posed

the

problem

the

more

we

need

to

make

assumptions

The

No

Free

Lunch

theorem

-LRB-

Wolpert

and

Macready

-RRB-

shows

that

if

we

do

not

make

any

assumptions

regarding

the

target

function

no

algorithm

performs

better

than

any

other

on

average

In

other

words

we

need

to

make

assumptions

A

very

convenient

and

useful

assumption

-LRB-

which

corresponds

to

common

sensory

data

in

many

instances

-RRB-

is

that

of

smoothness

As

demonstrated

smoother

functions

correspond

to

faster

convergence

Intuitively

this

is

clear

more

complex

functions

correspond

to

a

greater

degree

of

saturation

in

the

nodes

and

we

know

that

the

backpropagated

error

approaches

zero

in

saturated

regions

Universal

Approximation

The

results

are

negative

in

terms

of

the

possibility

of

training

large

homogeneous

MLPs

to

parsimoniously

represent

arbitrary

functions

Even

for

the

case

of

relatively

small

maximum

weights

in

the

network

we

can

see

that

convergence

can

be

very

slow

for

the

most

parsimonious

solutions

While

large

MLPs

have

been

very

successful

in

pattern

recognition

-LRB-

e.g.

speech

phoneme

recognition

-LRB-

Bourlard

and

Morgan

-RRB-

-RRB-

we

suggest

that

for

more

intelligent

behaviour

the

emergence

of

more

parsimonious

representations

of

computational

structures

is

required

and

this

is

dif?cult

with

backpropagation

With

reference

to

biological

neural

networks

the

reason

that

we

can

learn

the

things

we

do

is

perhaps

critically

linked

to

the

pre-wiring

of

our

brain

For

example

we

know

that

we

have

a

lot

of

dif?culty

training

a

chimpanzee

to

learn

our

language

let

alone

MLPs

We

note

that

this

conclusion

applies

to

the

homogeneous

MLP

type

of

universal

approximation

approach

to

learning

by

example

There

is

a

whole

class

of

algorithms

for

learning

by

example

which

operate

on

the

basis

of

looking

for

regularities

and

then

incorporating

these

regularities

into

the

model

-LRB-

e.g.

grammar

induction

algorithms

-RRB-

In

contrast

the

computational

capacity

of

MLPs

is

static

Finally

we

quote

Minsky

and

Papert

's

epilogue

to

Perceptrons

-LRB-

Minsky

and

Papert

-RRB-

In

the

early

years

of

cybernetics

everyone

understood

that

hill-climbing

was

always

available

for

working

easy

problems

but

that

it

almost

always

became

impractical

for

problems

of

larger

sizes

and

complexities

We

were

very

pleased

to

discover

that

-LSB-

perceptron

convergence

-RSB-

could

be

represented

as

hill-climbing

however

that

very

fact

led

us

to

wonder

whether

such

procedures

could

dependably

be

generalized

even

to

the

limited

class

of

multi-layer

machines

we

have

named

Gamba

perceptrons

The

situation

seems

not

to

have

changed

much

we

have

seen

no

contemporary

connectionist

publication

that

casts

much

new

theoretical

light

on

the

situation

Then

why

has

-LSB-

backpropagation

gradient

descent

-RSB-

become

so

popular

in

recent

years

In

part

this

is

because

it

is

so

widely

applicable

and

because

it

does

indeed

yield

new

results

-LRB-

at

least

on

problems

of

rather

small

scale

-RRB-

Its

reputation

also

gains

we

think

from

it

being

presented

in

forms

that

shares

albeit

to

a

lesser

degree

the

biological

plausibility

of

-LSB-

the

perceptron

-RSB-

But

we

fear

that

its

reputation

also

stems

from

unfamiliarity

with

the

manner

in

which

hill-climbing

methods

deteriorate

when

confronted

with

larger

scale

problems

Minsky

and

Papert

's

assertion

that

local

minima

and

related

dif?culties

are

a

problem

appears

to

be

valid

We

agree

standard

multi-layer

perceptrons

trained

with

backpropagation

are

not

the

solution

to

all

of

our

problems

However

while

there

are

certain

fundamental

limitations

to

the

performance

of

this

class

of

learning

algorithms

MLPs

have

produced

better

results

than

some

notable

alternatives

e.g.

perceptrons

with

threshold

units

The

imposition

of

the

sigmoid

non-linearities

in

MLPs

creates

an

error

surface

which

can

be

-LRB-

relatively

speaking

-RRB-

very

suitable

for

gradient

descent

based

approaches

Heuristic

minimization

are

able

to

take

advantage

of

this

to

nd

better

solutions

Acknowledgments

We

would

like

to

acknowledge

useful

discussions

with

R.

Carauna

S.

Hanson

B.G.

Horne

T.

Lin

K.-R

M?uller

and

T.

Rognvaldsson

This

work

has

been

partially

supported

by

the

Australian

Research

Council

-LRB-

ACT

-RRB-

and

the

Australian

Telecommunications

and

Electronics

Research

Board

-LRB-

SL

-RRB-

References

Abu-Mostafa

Y.

-LRB-

-RRB-

The

Vapnik-Chervonenkis

dimension

Information

versus

complexity

in

learning

Neural

Computation

-LRB-

-RRB-

Akaike

H.

-LRB-

-RRB-

Statistical

predictor

identi?cation

Annals

of

the

Institute

for

Statistical

Mathematics

Akaike

H.

-LRB-

-RRB-

Information

theory

and

an

extension

of

the

maximum

likelihood

principle

In

Petrov

B.

N.

and

Csaki

F.

editors

Proceeding

International

Symposium

on

Information

Theory

pages

Akademia

Kiado

Budapest

Akaike

H.

-LRB-

-RRB-

A

new

look

at

the

statistical

model

identi?cation

IEEE

Transactions

on

Automatic

Control

Amari

S.

-LRB-

-RRB-

Learning

and

statistical

inference

In

Arbib

M.

A.

editor

The

Handbook

of

Brain

Theory

and

Neural

Networks

pages

MIT

Press

Cambridge

Massachusetts

Baldi

P.

and

Hornik

K.

-LRB-

-RRB-

Neural

networks

and

principal

component

analysis

Learning

from

examples

without

local

minima

Neural

Networks

-LRB-

-RRB-

Barron

A.

R.

-LRB-

-RRB-

Complexity

regularization

with

application

to

arti?cial

neural

networks

In

Roussas

G.

editor

Nonparametric

Functional

Estimation

and

Related

Topics

pages

Kluwer

Academic

Publishers

Dordrecht

The

Netherlands

Barron

A.

R.

-LRB-

-RRB-

Neural

net

approximation

In

Proceedings

of

the

Seventh

Yale

Workshop

on

Adaptive

and

Learning

Systems

pages

Yale

University

New

Haven

CT.

Bartlett

P.

-LRB-

-RRB-

Vapnik-Chervonenkis

dimension

bounds

for

two-and

three-layer

networks

Neural

Computation

-LRB-

-RRB-

Baum

E.

and

Haussler

D.

-LRB-

-RRB-

What

size

net

gives

valid

generalization

Neural

Computation

-LRB-

-RRB-

Bourlard

H.

and

Morgan

N.

-LRB-

-RRB-

Connnectionist

Speech

Recognition

A

Hybrid

Approach

Kluwer

Academic

Publishers

Boston

MA

Ch

Statistical

Inference

in

MLPs

Caruana

R.

-LRB-

-RRB-

Generalization

vs.

Net

Size

Neural

Information

Processing

Systems

Tutorial

Denver

CO.

Cohn

D.

and

Tesauro

G.

-LRB-

-RRB-

How

tight

are

the

Vapnik-Chervonenkis

bounds

Neural

Computation

-LRB-

-RRB-

example

a

step

size

which

is

not

very

small

and

stochastic

update

Crane

R.

Fefferman

C.

Markel

S.

and

Pearson

J.

-LRB-

-RRB-

Characterizing

neural

network

error

surfaces

with

a

sequential

quadratic

programming

algorithm

In

Machines

That

Learn

Snowbird

Drucker

H.

Cortes

C.

Jackel

L.

Le

Cun

Y.

and

Vapnik

V.

-LRB-

-RRB-

Boosting

and

other

ensemble

methods

Neural

Computation

Friedman

J.

-LRB-

-RRB-

Introduction

to

computational

learning

and

statistical

prediction

In

Neural

Information

Processing

Systems

Tutorial

Denver

Colorado

Geman

S.

Bienenstock

E.

and

Doursat

R.

-LRB-

-RRB-

Neural

networks

and

the

bias/variance

dilemma

Neural

Computation

-LRB-

-RRB-

Gori

M.

and

Tesi

A.

-LRB-

-RRB-

On

the

problem

of

local

minima

in

backpropagation

IEEE

Transactions

on

Pattern

Analysis

and

Machine

Intelligence

-LRB-

-RRB-

Hamey

L.

-LRB-

-RRB-

Analysis

of

the

error

surface

of

the

XOR

network

with

two

hidden

nodes

Technical

Report

Macquarie

University

Sydney

Australia

Hassibi

B.

and

Stork

D.

G.

-LRB-

-RRB-

Second

order

derivatives

for

network

pruning

Optimal

brain

surgeon

In

Giles

C.

L.

Hanson

S.

J.

and

Cowan

J.

D.

editors

Advances

in

Neural

Information

Processing

Systems

volume

Morgan

Kaufmann

San

Mateo

CA

Haykin

S.

-LRB-

-RRB-

Neural

Networks

A

Comprehensive

Foundation

Macmillan

New

York

NY

Jacobs

R.

-LRB-

-RRB-

Methods

for

combining

experts

probability

assessments

Neural

Computation

Kohonen

T.

-LRB-

-RRB-

Self-Organizing

Maps

Springer-Verlag

Berlin

Germany

Le

Cun

Y.

-LRB-

-RRB-

Ef?cient

learning

and

second

order

methods

In

Advances

in

Neural

Information

Processing

Systems

Tutorial

Le

Cun

Y.

Denker

J.

and

Solla

S.

-LRB-

-RRB-

Optimal

brain

damage

In

Touretzky

D.

editor

Advances

in

Neural

Information

Processing

Systems

volume

pages

San

Mateo

-LRB-

Denver

-RRB-

Morgan

Kaufmann

Lorentz

G.

G.

-LRB-

-RRB-

Approximation

of

Functions

Holt-Rinehart-Winston

New

York

Maass

W.

-LRB-

-RRB-

Vapnik-Chervonenkis

dimension

of

neural

networks

In

Arbib

M.

A.

editor

The

Handbook

of

Brain

Theory

and

Neural

Networks

pages

MIT

Press

Cambridge

Massachusetts

McInerny

J.

Haines

K.

Biafore

S.

and

Hecht-Nielsen

R.

-LRB-

-RRB-

Back

propagation

error

surfaces

can

have

local

minima

In

International

Joint

Conference

on

Neural

Networks

volume

page

New

York

-LRB-

Washington

-RRB-

IEEE

Mhaskar

H.

-LRB-

-RRB-

Approximation

properties

of

a

multi-layer

feedforward

arti?cial

neural

network

Advances

in

Computational

Mathematics

Minsky

M.

and

Papert

S.

-LRB-

-RRB-

Perceptrons

MIT

Press

Cambridge

MA

expanded

version

of

the

original

edition

Moody

J.

-LRB-

-RRB-

The

effective

number

of

parameters

An

analysis

of

generalization

and

regularization

in

nonlinear

learning

systems

In

Moody

J.

Hanson

S.

J.

and

Lippmann

R.

P.

editors

Advances

in

Neural

Information

Processing

Systems

volume

pages

Morgan

Kaufmann

San

Mateo

CA

M?uller

K.

Finke

M.

Schulten

K.

Murata

N.

and

Amari

S.

-LRB-

-RRB-

A

numerical

study

on

learning

curves

in

stochastic

multi-layer

feed-forward

networks

Neural

Computation

in

press

Naftaly

U.

Intrator

N.

and

Horn

D.

-LRB-

-RRB-

Training

single

networks

for

optimal

ensemble

performance

In

Gerald

Tesauro

David

Touretzky

T.

L.

editor

Advances

in

Neural

Information

Processing

Systems

Cambridge

Massachusetts

MIT

Press

Ripley

B.

D.

-LRB-

-RRB-

Statistical

ideas

for

selecting

network

architectures

In

Invited

Presentation

Neural

Information

Processing

Systems

volume

Saad

D.

and

Solla

S.

-LRB-

-RRB-

Exact

solution

for

on-line

learning

in

multilayer

neural

networks

Phys

Rev.

Lett.

Saad

D.

and

Solla

S.

A.

-LRB-

-RRB-

Learning

from

corrupted

examples

in

multilayer

networks

Technical

Report

Aston

University

UK

Sartori

M.

A.

and

Antsaklis

P.

J.

-LRB-

-RRB-

A

simple

method

to

derive

bounds

on

the

size

and

to

train

multilayer

neural

networks

IEEE

Transactions

on

Neural

Networks

Vapnik

V.

N.

-LRB-

-RRB-

The

Nature

of

Statistical

Learning

Theory

Springer

Wolpert

D.

H.

and

Macready

W.

G.

-LRB-

-RRB-

No

free

lunch

theorems

for

search

Technical

Report

The

Santa

Fe

Institute

Yu

X.-H

-LRB-

-RRB-

Can

backpropagation

error

surface

not

have

local

minima

IEEE

Transactions

on

Neural

Networks

O O O O O O B O B O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O B O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O B O O O O O O O B I O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O B I O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B I O O O O O O O O O O O

backpropagation
committees
convergence
curse of dimensionality
ensembles
function approximation
generalization
local minima
network size
smoothness