An

Analysis

of

Noise

in

Recurrent

Neural

Networks

Convergence

and

Generalization

Kam-Chuen

Jim

C.

Lee

Giles

Bill

G.

Horne

NEC

Research

Institute

Inc.

Independence

Way

Princeton

NJ

fkamjim,giles,horneg@research.nj.nec.com

Also

with

Institute

for

Advanced

Computer

Studies

University

of

Maryland

College

Park

MD

Abstract

There

has

been

much

interest

in

applying

noise

to

feedforward

neural

networks

in

order

to

observe

their

effect

on

network

performance

We

extend

these

results

by

introducing

and

analyzing

various

methods

of

injecting

synaptic

noise

into

dynamically-driven

recurrent

networks

during

training

We

present

theoretical

results

which

show

that

applying

a

controlled

amount

of

noise

during

training

may

improve

convergence

and

generalization

performance

In

addition

we

analyze

the

effects

of

various

noise

parameters

-LRB-

additive

vs.

multiplicative

cumulative

vs.

non-cumulative

per

time

step

vs.

per

string

-RRB-

and

predict

that

best

overall

performance

can

be

achieved

by

injecting

additive

noise

at

each

time

step

Noise

contributes

a

second-order

gradient

term

to

the

error

function

which

can

be

viewed

as

an

anticipatory

agent

to

aid

convergence

This

term

appears

to

find

promising

regions

of

weight

space

in

the

beginning

stages

of

training

when

the

training

error

is

large

and

should

improve

convergence

on

error

surfaces

with

local

minima

The

first

order

term

can

be

interpreted

as

a

regularization

term

that

can

improve

generalization

Specifically

this

term

can

encourage

internal

representations

where

the

state

nodes

operate

in

the

saturated

region

of

the

sigmoid

discriminant

function

While

this

effect

can

improve

performance

on

automata

inference

problems

with

binary

inputs

and

target

outputs

it

is

unclear

what

effect

it

will

have

on

other

types

of

problems

To

substantiate

these

predictions

we

present

simulations

on

learning

the

dual

parity

grammar

from

temporal

strings

for

all

noise

models

and

present

simulations

on

learning

a

randomly

generated

grammar

using

the

predicted

best

noise

model

Printed

in

IEEE

Transactions

on

Neural

Networks

vol

no.

Copyright

IEEE

INTRODUCTION

There

has

been

a

great

deal

of

work

on

enhancing

neural

network

performance

Two

important

performance

parameters

are

convergence

and

generalization

Convergence

is

the

amount

of

time

measured

in

CPU

operations

or

training

epochs

required

to

find

an

acceptable

solution

during

training

Generalization

measures

the

ability

to

correctly

classify

new

unseen

data

Among

some

of

the

important

methods

that

have

been

introduced

to

improve

convergence

is

the

notion

of

searching

the

coarse

regions

of

state

space

before

searching

finer

regions

Simulated

annealing

is

one

such

method

and

was

shown

by

Kirkpatrick

et.al

-LSB-

-RSB-

to

be

very

effective

on

combinatorial

problems

like

graph

partitioning

Similarly

Ackley

et.al

and

Hinton

and

Sejnowski

-LSB-

-RSB-

used

noise

in

Boltzmann

machines

to

escape

from

local

minima

Specifically

it

was

indicated

that

at

high

noise

temperatures

Boltzmann

machines

will

search

the

overall

structure

of

state

space

for

a

coarse

minimum

while

subsequently

lowering

the

noise

temperature

will

allow

a

better

minimum

to

be

found

within

the

coarse-scale

minimum

It

has

been

shown

that

generalization

can

be

improved

by

removing

the

redundancies

and

irrelevancies

in

a

network

There

have

been

at

least

two

approaches

pruning

and

regularization

An

example

of

pruning

is

optimal

brain

damage

which

can

improve

generalization

ability

and

speed

of

learning

by

using

secondderivative

information

to

remove

unimportant

weights

from

the

network

-LRB-

Cun

Denker

and

Solla

-LSB-

-RSB-

-RRB-

Also

Giles

and

Omlin

-LSB-

-RSB-

have

demonstrated

improvement

in

generalization

of

recurrent

neural

networks

after

pruning

Other

pruning

methods

can

be

found

in

Reed

-LSB-

-RSB-

Weight

decay

is

an

example

of

a

regularization

method

and

was

shown

by

Krogh

and

Hertz

-LSB-

-RSB-

to

improve

generalization

on

feed-forward

networks

by

suppressing

irrelevant

components

of

the

weight

vector

Previous

Work

on

Training

with

Noise

Previous

research

has

investigated

the

effects

of

noise

on

feedforward

neural

networks

Training

with

noise

can

lead

to

more

realistic

biological

models

For

example

Hinton

Sejnowski

and

Ackley

-LSB-

-RSB-

compared

noise

temperature

in

the

Boltzmann

machine

to

the

addition

of

Gaussian

noise

at

neuronal

membrane

potentials

Bremermann

and

Anderson

-LSB-

-RSB-

introduced

a

biased

random

walk

in

weight

space

-LRB-

the

chemotaxis

algorithm

-RRB-

as

a

biologically

plausible

learning

rule

and

Kilis

and

Ackerman

-LSB-

-RSB-

introduced

a

stochastic

neuron

model

which

uses

Monte

Carlo

techniques

to

stochastically

bias

the

firing

decision

of

neurons

More

relevant

to

this

paper

however

are

the

experiments

that

use

noise

injection

during

training

to

enhance

network

performance

Much

work

in

training

with

noise

has

been

motivated

by

the

desire

to

improve

fault

tolerance

i.e.

the

graceful

degradation

of

performance

in

the

presence

of

faults

Judd

and

Munro

-LSB-

-RSB-

showed

that

using

noisy

hidden

nodes

during

training

can

result

in

error-correcting

codes

which

increase

the

tolerance

of

feedforward

nets

to

unreliable

nodes

Similarly

S?equin

and

Clay

-LSB-

-RSB-

showed

that

randomly

disabling

hidden

nodes

-LRB-

i.e.

clamp

to

zero

-RRB-

during

the

training

phase

increases

the

tolerance

of

multilayer

perceptrons

to

node

failures

Minnix

-LSB-

-RSB-

also

demonstrated

improved

fault

tolerance

by

training

on

noisy

inputs

and

argued

that

such

a

method

helps

prevent

pattern

memorization

Murray

and

Edwards

-LSB-

-RSB-

utilized

synaptic

noise

during

training

to

improve

fault

tolerance

and

training

quality

Omlin

and

Giles

-LSB-

-RSB-

have

extended

the

fault

tolerance

considerations

to

recurrent

neural

networks

With

the

exception

of

Murray

and

Edwards

-LSB-

-RSB-

these

methods

generally

indicate

convergence

as

a

trade-off

for

enhanced

performance

Recently

stochastic

techniques

such

as

Nodal

Perturbation

Weight

Perturbation

and

Summed

Weight

Neuron

Perturbation

have

been

introduced

which

in

some

cases

may

converge

more

quickly

than

pure

gradient

descent

-LRB-

Cauwenberghs

-LSB-

-RSB-

Dembo

and

Kailath

-LSB-

-RSB-

Flower

and

Jabri

-LSB-

-RSB-

-RRB-

These

methods

in

general

require

more

epochs

to

converge

but

less

computation

per

epoch

mainly

because

calculation

of

the

true

gradient

is

not

required

Also

it

has

been

demonstrated

that

faster

learning

rates

can

be

achieved

by

adding

noise

to

the

weight

updates

and

adapting

the

magnitude

of

such

noise

to

the

output

error

-LRB-

Burton

and

Mpitsos

-LSB-

-RSB-

-RRB-

Hanson

developed

a

stochastic

version

of

the

delta

rule

which

adapt

weight

means

and

standard

deviations

instead

of

clean

weight

values

and

demonstrated

improved

convergence

rates

-LSB-

-RSB-

Training

with

noise

can

also

improve

generalization

Holmstr?om

and

Koistinen

-LSB-

-RSB-

showed

that

training

with

additive

input

noise

can

improve

generalization

in

feedforward

networks

and

provided

mathematically

justified

rules

for

choosing

the

characteristics

of

the

noise

Bishop

-LSB-

-RSB-

showed

that

training

with

noisy

data

is

equivalent

to

Tikhonov

Regularization

and

suggested

directly

minimizing

the

regularized

error

function

as

a

practical

alternative

Overview

of

paper

To

date

there

has

been

little

work

on

applying

noise

insertion

methods

to

recurrent

neural

networks

In

this

paper

we

explore

the

effects

of

synaptic

noise

on

recurrent

neural

networks

We

introduce

several

ways

to

inject

noise

during

training

and

demonstrate

that

utilizing

noise-enhanced

error

functions

can

improve

convergence

and

generalization

Previous

work

on

improving

these

two

performance

measures

as

discussed

above

focused

on

simplifying

feedforward

neural

networks

and

on

ways

of

first

searching

the

coarse

regions

of

state

space

The

function

of

synaptic

noise

in

recurrent

nets

is

novel

in

the

sense

that

synaptic

noise

can

improve

convergence

by

searching

for

promising

regions

of

state

space

and

at

the

same

time

enhance

generalization

by

favoring

saturated

state

nodes

Promising

regions

of

state

space

do

not

necessarily

correspond

to

coarsely-spaced

local

minima

and

saturated

state

nodes

are

in

general

different

from

a

simplified

network

Besides

an

analysis

of

noise

in

recurrent

neural

networks

we

perform

many

simulations

to

verify

our

analysis

For

recurrent

neural

network

and

problem

selection

we

use

a

second-order

fully-connected

recurrent

neural

network

This

recurrent

neural

network

is

then

trained

to

learn

automata

from

grammatical

strings

encoded

as

temporal

sequences

For

a

small

neural

network

and

small

automata

we

ran

many

simulations

and

thus

show

statistical

confidence

in

our

results

The

next

section

section

defines

the

second-order

fully-connected

recurrent

neural

network

used

in

simulations

and

introduces

several

methods

of

implementing

synaptic

noise

We

discuss

different

noise

models

and

compare

their

effects

on

the

error

function

in

section

From

this

analysis

predictions

are

made

regarding

their

effects

on

convergence

and

generalization

performance

Section

presents

simulation

results

on

learning

the

dual

parity

and

a

small

randomly

generated

automata

from

strings

encoded

as

temporal

sequences

These

results

are

general

to

all

incremental

weight

updating

methods

and

should

extend

easily

to

other

recurrent

networks

Conclusions

and

open

questions

follow

in

section

NOISE

INJECTION

IN

RECURRENTNEURAL

NETWORKS

Noiseless

Recurrent

Network

Architecture

and

Training

In

this

paper

we

restrict

ourselves

to

second-order

fully-connected

discrete-time

recurrent

networks

since

such

networks

have

demonstrated

good

performance

on

the

types

of

problems

we

discuss

in

this

paper

namely

that

of

learning

finite

state

grammars

-LSB-

-RSB-

The

discrete-time

recurrent

network

architecture

consists

of

N

recurrent

state

neurons

Sj

and

L

non-recurrent

input

neurons

Ik

The

states

are

related

by

a

set

of

weights

Wijk

The

recurrent

network

operation

resembles

that

of

a

finite

state

machine

with

the

state

process

finput

current

stateg

=-RRB-

fnext

stateg

as

defined

by

the

following

equations

St

i

g

-LRB-

t

i

-RRB-

-LRB-

-RRB-

t

i

X

j

k

WijkStjItk

-LRB-

-RRB-

where

i

is

the

net

input

to

unit

i

and

g

-LRB-

x

-RRB-

is

a

predetermined

sigmoid

discriminant

function

In

the

simulations

described

in

section

g

-LRB-

x

-RRB-

is

defined

as

g

-LRB-

x

-RRB-

e?x

-LRB-

-RRB-

One-hot

or

unary

input

encoding

is

used

The

input

symbols

are

encoded

as

orthonormal

unit

vectors

in

L-dimensional

space

so

the

inputs

are

in

the

form

Ik

ffikff

for

-LRB-

ff

L

-RRB-

For

the

dual

parity

DFA

L

in

order

to

represent

the

binary

input

characters

and

the

empty

string

e.

Hence

the

possible

symbols

are

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

Each

input

string

is

presented

to

the

network

one

character

per

discrete

time

step

t

At

time

t

T

when

the

entire

input

string

has

been

presented

to

the

network

the

value

at

a

selected

neuron

SO

is

defined

to

be

the

output

An

input

string

is

accepted

if

SO

ffl

or

rejected

if

SO

ffl

where

ffl

is

the

error

tolerance

and

is

a

positive

real

number

Weights

are

updated

at

the

end

of

each

string

During

training

an

error

function

is

computed

as

Ep

-LRB-

STO

dp

-RRB-

-LRB-

-RRB-

where

dp

is

the

target

output

value

for

pattern

p

-LRB-

either

or

depending

on

the

DFA

response

of

the

input

string

-RRB-

and

fflp

is

the

raw

error

For

complexity

reasons

we

used

the

Back-Propagation-Through-Time

-LRB-

BPTT

-RRB-

training

algorithm

-LSB-

-RSB-

Incremental

learning

as

described

in

Giles

et.al

-LSB-

-RSB-

is

used

to

descend

the

error

function

Basically

the

network

is

trained

on

a

subset

of

the

training

set

called

the

working

set

which

gradually

increases

in

size

until

the

network

is

able

to

correctly

classify

the

entire

training

set

Strings

from

the

working

set

are

presented

to

the

network

in

alphabetical

order

Incremental

learning

reduces

the

training

time

required

because

often

not

all

the

strings

in

the

training

set

need

to

be

seen

before

the

network

can

classify

the

entire

set

correctly

Training

stops

when

all

strings

in

the

training

set

are

correctly

classified

Noise

Injection

Murray

and

Edwards

-LSB-

-RSB-

simulated

synaptic

noise

on

multilayered

perceptrons

-LRB-

MLPs

-RRB-

by

applying

noise

to

the

weights

of

each

layer

during

training

Applying

this

method

to

recurrent

networks

is

not

straightforward

because

effectively

the

same

weights

are

propagated

forward

in

time

This

can

be

seen

by

recalling

the

BPTT

representation

of

unrolling

a

recurrent

network

in

time

into

T

layers

with

identical

weights

where

T

is

the

length

of

the

temporal

sequence

As

in

Burton

and

Mpitsos

-LSB-

-RSB-

Hanson

-LSB-

-RSB-

and

Murray

and

Edwards

-LSB-

-RSB-

we

will

only

focus

on

noise

injection

in

synaptic

weights

and

ignore

other

methods

of

noise

insertion

Because

of

its

temporal

nature

synaptic

noise

injection

of

recurrent

networks

introduces

two

issues

which

do

not

pertain

to

feedforward

nets

What

is

the

frequency

of

noise

injection

Is

the

noise

injection

cumulative

or

non-cumulative

This

is

a

direct

result

of

the

fact

that

recurrent

networks

effectively

propagate

the

same

weights

forward

in

time

Should

one

set

of

noise

values

also

be

propagated

forward

or

should

a

different

set

of

noise

values

be

applied

at

each

time

step

Or

should

noise

be

applied

only

at

the

beginning

of

each

input

sequence

If

noise

is

propagated

forward

in

time

should

it

accumulate

in

the

weights

over

all

time

steps

or

can

the

noise

be

inserted

independently

each

time

step

In

this

paper

we

consider

the

cases

where

noise

is

propagated

forward

in

time

We

introduce

a

per

time

step

method

of

injecting

synaptic

noise

into

recurrent

networks

which

inserts

a

new

random

noise

value

at

each

time

step

of

the

forward

pass

during

training

On

the

other

hand

a

per

string

model

can

be

viewed

as

a

special

case

of

the

per

time

step

model

such

that

the

same

set

of

noise

values

is

injected

for

all

time

steps

of

each

given

input

string

Non-cumulative

models

apply

a

new

noise

value

ijk

to

a

noise-free

weight

at

each

instance

of

noise

injection

while

cumulative

models

apply

a

new

noise

to

a

previously

perturbed

weight

at

each

instance

Non-cumulative

noise

can

be

represented

as

Wt

ijk

N

-LRB-

W

ij

k

-RRB-

-LRB-

-RRB-

where

fW

ij

kg

is

the

noise-free

weight

set

and

N

is

the

noise

process

applied

to

weight

W

ij

k.

Cumulative

noise

be

represented

by

ijk

N

-LRB-

W

ij

k

-RRB-

-LRB-

-RRB-

Wt

ijk

N

-LRB-

Wt

ijk

-RRB-

t

-LRB-

-RRB-

An

issue

common

to

both

feedforward

and

recurrent

networks

is

whether

noise

is

additive

or

multiplicative

Additive

noise

perturbs

a

weight

parameter

such

that

Wijk

Wijk

ijk

-LRB-

-RRB-

per

time

step

non-cumulative

W

W

W

per

time

step

cumulative

W

W

W

per

string

non-cumulative

W

W

W

per

string

cumulative

W

W

W

Table

Sample

noise

injection

steps

for

all

additive

noise

models

at

time

steps

and

W

is

the

noise-free

weight

's

are

the

noise

terms

for

each

time

step

where

ijk

is

a

random

noise

value

On

the

other

hand

multiplicative

noise

perturbs

each

weight

such

that

Wijk

Wijk

Wijk?ijk

-LRB-

-RRB-

Table

clarifies

these

issues

by

illustrating

the

noise

injection

steps

for

all

additive

noise

processes

The

's

are

the

various

noise

terms

and

W

is

the

noise

free-weight

For

per

time

step

non-cumulative

noise

insertion

a

different

noise

is

added

at

each

time

step

and

the

noise

from

the

previous

time

step

does

not

accumulate

in

time

For

the

per

time

step

cumulative

noise

insertion

all

previously

added

noise

is

carried

over

to

the

next

time

step

For

per

string

-LRB-

or

per

sequence

-RRB-

non-cumulative

the

same

noise

is

inserted

at

the

beginning

and

used

throughout

the

temporal

updates

For

per

string

cumulative

the

initial

noise

accumulates

for

each

time

step

Similar

results

would

be

seen

for

the

multiplicative

noise

case

We

report

experiments

which

explore

all

combinations

of

additive

vs.

multiplicative

cumulative

vs.

noncumulative

and

per

string

vs.

per

time

step

noise

models

In

all

noise

models

the

actual

weight

updates

Wijk

's

descend

the

noise-enhanced

error

term

Ep

These

updates

are

applied

to

the

noise-free

weight

set

This

prevents

noisy

updates

allows

simpler

mathematical

analysis

and

for

the

purpose

of

these

experiments

has

the

desirable

effect

of

limiting

the

effect

of

noise

to

the

error

function

For

all

cases

the

noise

source

is

chosen

to

be

a

random

zero-mean

white

noise

process

uniformly

distributed

between

-LRB-

a

a

-RRB-

where

a

is

a

positive

constant

parameter

ANALYSIS

OF

SYNAPTIC

NOISE

The

effects

of

each

noise

model

is

analyzed

by

taking

the

Taylor

expansion

on

the

error

function

around

the

noise-free

weight

set

By

truncating

this

expansion

to

second

order

and

lower

terms

we

can

interpret

the

effect

of

noise

as

a

set

of

regularization

terms

applied

to

the

error

function

From

these

terms

predictions

can

be

made

about

the

effects

on

generalization

and

convergence

A

similar

analysis

was

performed

by

Murray

and

Edwards

-LSB-

-RSB-

on

MLPs

to

demonstrate

the

effects

of

synaptic

noise

on

fault

tolerance

and

training

quality

Le

Cun

et

al.

-LSB-

-RSB-

also

applied

a

Taylor

expansion

on

the

error

function

to

approximate

the

effects

NON-CUMULATIVE

Additive

Multiplicative

Noise

step

Wt

ijk

W

ij

k

t

ijk

Wt

ijk

W

ij

k

W

ij

k?t

ijk

order

T

X

t

X

ijk

@STO

@Wt

ijk

T

X

t

X

ijk

Wt

ijk

@STO

@Wt

ijk

order

T

X

t

X

ijk

-LRB-

Wt

ijk

-RRB-

fflp

T

X

t

X

ijk

-LRB-

Wt

ijk

-RRB-

-LRB-

Wt

ijk

-RRB-

CUMULATIVE

Additive

Multiplicative

Noise

step

Wt

ijk

W

ij

k

t

X

o

o

ijk

Wt

ijk

W

ij

k

tY

o

-LRB-

o

ijk

-RRB-

order

T

X

t

u

X

ijk

v

@STO

@Wt

ijk

@STO

@Wu

ijk

T

X

t

u

X

ijk

vWt

ijkWu

ijk

@STO

@Wt

ijk

@STO

@Wu

ijk

order

T

X

t

u

X

ijk

v

@Wt

ijk@Wu;ijk

T

X

t

u

X

ijk

vWt

ijkWu

ijk

@Wt

ijk@Wu;ijk

Table

Per

time

step

noise

injection

terms

and

error

function

expansion

terms

for

cumulative

and

noncumulative

noise

for

both

the

additive

and

multiplicative

cases

v

min

-LRB-

t

u

-RRB-

W

is

the

noise-free

weight

set

of

small

weight

perturbations

and

dropped

the

first

order

terms

by

assuming

the

network

is

at

convergence

As

in

-LSB-

-RSB-

we

do

not

assume

the

networks

are

at

convergence

and

consider

both

first

and

second

order

terms

Tables

and

list

the

noise

injection

step

and

resulting

first

and

second

order

Taylor

expansion

terms

for

all

noise

models

These

terms

are

derived

in

Appendix

A

Our

results

are

comparable

-LRB-

in

terms

of

the

effects

on

the

error

function

-RRB-

to

feedforward

nets

only

when

the

recurrent

noise

models

are

non-cumulative

and

per

time-step

Cumulative

noise

and

per

string

noise

in

recurrent

nets

are

not

comparable

to

their

feedforward

versions

because

of

the

absence

of

error-weight

correlations

across

time

in

feedforward

nets

In

feedforward

nets

per

string

noise

is

identical

to

per

time

step

noise

because

the

length

of

each

pattern

in

time

is

always

one

Also

cumulative

noise

in

feedforward

networks

are

identical

to

noncumulative

noise

when

the

weights

are

updated

after

the

presentation

of

each

training

sample

but

have

different

variance

when

using

batch

training

-LRB-

see

Appendix

B

-RRB-

Thus

our

noncumulative

multiplicative

per

time

step

analytical

results

are

consistent

with

Murray

and

Edwards

-LSB-

-RSB-

who

applied

multiplicative

synaptic

noise

to

MLPs

Other

noise

models

accumulate

noise

at

different

rates

and

have

different

convergence

and

generalization

effects

as

described

below

and

summarized

in

Tables

and

NON-CUMULATIVE

Additive

Multiplicative

Noise

step

Wt

ijk

W

ij

k

ijk

Wt

ijk

W

ij

k

W

ij

k?ijk

order

T

X

t

u

X

ijk

@STO

@Wt

ijk

@STO

@Wu

ijk

T

X

t

u

X

ijk

Wt

ijkWu

ijk

@STO

@Wt

ijk

@STO

@Wu

ijk

order

T

X

t

u

X

ijk

@Wt

ijk@Wu;ijk

T

X

t

u

X

ijk

Wt

ijkWu

ijk

@Wt

ijkWu

ijk

CUMULATIVE

Additive

Multiplicative

Noise

step

Wt

ijk

W

ij

k

-LRB-

t

-RRB-

ijk

Wt

ijk

W

ij

k

-LRB-

ijk

-RRB-

t

order

T

X

t

u

X

ijk

-LRB-

t

-RRB-

-LRB-

u

-RRB-

@STO

@Wt

ijk

@STO

@Wu

ijk

T

X

t

u

X

ijk

-LRB-

t

-RRB-

-LRB-

u

-RRB-

@STO

@Wt

ijk

@STO

@Wu

ijk

T

X

t

X

ijk

t

-LRB-

t

-RRB-

@STO

@Wt

ijk

order

T

X

t

u

X

ijk

-LRB-

t

-RRB-

-LRB-

u

-RRB-

@Wt

ijk@Wu;ijk

T

X

t

u

X

ijk

-LRB-

t

-RRB-

-LRB-

u

-RRB-

@wt

ijk@Wu;ijk

Table

Per

string

noise

injection

terms

and

error

function

expansion

terms

for

cumulative

and

noncumulative

noise

for

both

the

additive

and

multiplicative

cases

W

is

the

noise-free

weight

set

Generalization

Improvement

due

to

Synaptic

Noise

We

propose

that

two

ways

to

improve

generalization

are

ffl

Reduce

the

raw

error

and

ffl

Improve

the

internal

representation

of

information

in

the

network

Reducing

the

raw

error

fflp

improves

the

tolerance

of

the

network

to

perturbations

in

the

output

nodes

This

tolerance

provides

a

buffer

from

which

slight

variations

in

the

output

node

will

lead

to

the

same

classification

thus

improving

generalization

At

the

same

time

it

has

been

shown

that

on

networks

with

too

many

parameters

over-training

can

lead

to

poor

generalization

since

reducing

the

raw

error

to

zero

causes

the

network

to

effectively

memorize

the

training

set

-LSB-

-RSB-

One

common

cause

of

poor

generalization

in

recurrent

networks

that

are

trained

on

hidden-state

problems

is

the

presence

of

unsaturated

state

representations

-LSB-

-RSB-

Typically

a

network

can

not

revisit

the

exact

same

point

in

state

space

but

tends

to

wander

away

from

its

learned

state

representation

One

approach

to

alleviate

this

problem

is

to

encourage

state

nodes

to

operate

in

the

saturated

regions

of

the

sigmoid

The

first

order

error

expansion

terms

of

most

noise

models

considered

are

capable

of

encouraging

the

network

to

achieve

saturated

states

This

can

be

shown

by

applying

the

chain

rule

to

the

partial

derivative

in

the

first

order

expansion

terms

@STO

@Wt

ijk

@STO

TO

X

l

TO

@ST

l

@ST

l

T

l

X

n

t

m

@Stn

@Stn

@Wt

ijk

-LRB-

-RRB-

where

ti

is

the

net

input

to

state

node

i

at

time

step

t

The

partial

derivatives

@S

favor

internal

representations

such

that

the

effects

of

perturbations

at

the

net

inputs

ti

on

the

states

Sti

are

minimized

While

this

effect

may

improve

performance

on

automata

inference

problems

-LRB-

learning

automata

from

temporal

sequences

of

strings

-RRB-

with

binary

inputs

and

target

outputs

it

is

unclear

what

effect

it

will

have

on

other

types

of

problems

Multiplicative

noise

implements

a

form

of

weight

decay

because

the

error

expansion

terms

include

the

weight

products

W

ijk

or

Wt

ijkWu

ijk

Although

weight

decay

has

been

shown

to

improve

generalization

on

feedforward

networks

-LRB-

Krogh

and

Hertz

-RRB-

we

hypothesize

that

in

general

weight

decay

will

not

always

improve

generalization

for

recurrent

networks

that

are

learning

finite

state

automata

-LRB-

FSA

-RRB-

problems

Large

weights

are

necessary

to

saturate

the

state

nodes

to

the

upper

and

lower

limits

of

the

sigmoid

discriminant

function

Therefore

we

predict

additive

noise

will

allow

better

generalization

on

automata

inference

problems

because

it

is

not

a

weight

decay

term

while

multiplicative

noise

will

work

better

on

other

types

of

problems

where

weight

decay

is

appropriate

Noise

models

whose

first

order

error

term

contain

the

expression

@STO

@Wt

ijk

@STO

@Wu

lmn

will

favor

saturated

states

for

those

partials

whose

sign

correspond

to

the

sign

of

a

majority

of

all

partials

These

noise

models

will

favor

unsaturated

states

operating

in

the

linear

region

of

the

sigmoid

for

partials

whose

sign

is

in

the

minority

Such

sign-dependence

is

not

optimal

because

not

all

the

weights

are

encouraged

to

be

saturated

The

error

terms

for

cumulative

per

time

step

noises

sum

a

product

with

the

expression

v

@STO

@Wt

ijk

@STO

@Wu

lmn

where

v

min

-LRB-

t

u

-RRB-

The

effect

of

cumulative

noise

increases

more

rapidly

because

of

v

and

thus

optimal

generalization

and

detrimental

noise

effects

will

occur

at

lower

amplitudes

than

non-cumulative

noise

For

cumulative

per

string

noise

models

the

products

-LRB-

t

-RRB-

-LRB-

u

-RRB-

and

t

ijk?u

lmn

in

the

expansion

terms

rapidly

overwhelm

the

raw

error

term

Generalization

improvement

is

not

expected

for

these

models

We

also

reason

that

all

generalization

enhancements

will

be

valid

only

for

a

range

of

noise

values

above

which

noise

overwhelms

the

raw

error

information

The

differences

between

the

noise

models

are

summarized

in

Table

where

a

check

indicates

that

the

corresponding

enhancement

is

predicted

It

is

important

to

realize

that

although

all

noise

models

are

theoretically

capable

of

improving

convergence

and

generalization

by

searching

for

promising

weights

and

saturated

states

Table

lists

the

additional

enhancements

to

the

Generalization

Convergence

Generalization

Convergence

Improvement

Improvement

Improvement

Sign-independent

favoring

Anticipatory

No

additional

weight

terms

of

saturated

states

effect

Per

Timestep

Cumulative

Additive

p

p

p

Multiplicative

Per

Timestep

Non-cumulative

Additive

p

p

p

Multiplicative

p

p

Per

String

Cumulative

Additive

p

p

Multiplicative

Per

String

Non-cumulative

Additive

p

p

Multiplicative

Table

Generalization

and

Convergence

Improvement

Checks

indicate

additional

enhancements

to

the

search

for

saturated

states/promising

weights

due

to

each

noise

model

Sign-independent

enforcement

of

saturated

states

improves

generalization

-LRB-

column

-RRB-

The

anticipatory

effect

is

guaranteed

only

when

there

are

either

no

weight

terms

in

the

second

order

derivatives

of

Tables

and

or

the

weight

terms

are

always

positive

-LRB-

column

-RRB-

The

absence

of

weight

terms

in

Tables

and

improves

convergence/generalization

by

not

constraining

the

weights

to

small

amplitudes

-LRB-

column

-RRB-

search

From

this

table

it

can

be

inferred

that

additive

per

time

step

noise

models

should

yield

the

best

generalization

performance

Convergence

Improvement

due

to

Synaptic

Noise

Convergence

can

be

represented

by

the

number

of

epochs

required

to

learn

the

training

set

This

measure

is

dependent

on

the

error

trajectory

followed

during

the

training

phase

On

error

surfaces

filled

with

local

minima

many

epochs

may

be

required

before

the

training

set

is

learned

because

the

learning

trajectory

may

enter

an

unacceptable

local

minima

and

face

a

difficult

or

impossible

task

of

exiting

from

it

Intuitively

synaptic

noise

may

alleviate

this

problem

by

allowing

the

network

to

make

noisy

jumps

out

of

local

minima

Obviously

this

will

be

beneficial

only

to

learning

problems

whose

error

surfaces

contain

local

minima

On

simple

error

surfaces

synaptic

noise

may

worsen

generalization

performance

by

perturbing

a

straightforward

training

trajectory

which

would

have

otherwise

led

directly

to

a

solution

Synaptic

noise

can

improve

convergence

by

favoring

promising

weights

in

the

beginning

stages

of

training

This

can

be

demonstrated

by

examining

the

second

order

error

expansion

term

for

non-cumulative

multiplicative

per

time

step

noise

T

X

t

X

ijk

-LRB-

Wt

ijk

-RRB-

-LRB-

Wt

ijk

-RRB-

When

fflp

is

negative

solutions

with

a

negative

second

order

state-weight

partial

derivative

will

be

destabilized

In

other

words

when

the

output

STO

is

too

small

the

network

will

favor

updating

in

a

direction

such

that

the

first

order

partial

derivative

is

increasing

A

corresponding

relationship

can

be

observed

for

the

case

when

fflp

is

positive

Thus

the

second

order

term

of

the

error

function

will

allow

a

higher

raw

error

fflp

to

be

favored

if

such

an

update

will

place

the

weights

in

a

more

promising

area

i.e.

a

region

where

weight

changes

are

likely

to

move

STO

in

a

direction

to

reduce

the

raw

error

The

anticipatory

effect

of

this

term

or

look-ahead

property

as

described

in

Murray

and

Edwards

-LSB-

-RSB-

is

more

important

in

the

beginning

stages

of

training

where

fflp

is

large

and

will

become

insignificant

in

the

finishing

stages

of

training

as

fflp

approaches

zero

In

general

the

second

order

terms

of

all

noise

models

as

shown

in

Tables

and

are

capable

of

this

anticipatory

effect

In

all

noise

models

these

terms

will

favor

updating

the

weights

such

that

the

sum

of

the

dependencies

of

the

output

node

on

all

weights

has

the

opposite

sign

of

the

raw

error

term

fflp

In

other

words

either

a

majority

of

the

weights

are

moved

to

a

more

promising

space

or

a

minority

of

the

weights

have

moved

to

a

more

promising

space

with

very

steep

slopes

Non-cumulative

multiplicative

per

time

step

noise

includes

the

squared

terms

W

ijk

which

are

always

positive

Thus

this

noise

model

and

all

additive

noise

models

are

guaranteed

to

have

this

anticipatory

effect

All

other

multiplicative

noise

models

have

the

products

Wt

ijkWu

ijk

or

Wt

ijkWu

lmn

implying

that

the

anticipatory

effect

is

observed

only

when

the

majority

of

the

weights

are

positive

This

is

summarized

in

Table

column

Similar

to

arguments

in

Section

the

absence

of

weight

decay

will

make

the

learning

task

easier

and

improve

convergence

-LRB-

see

Table

column

-RRB-

SIMULATION

RESULTS

Experimental

Methodology

Two

sets

of

experiments

which

explore

the

effects

of

all

combinations

of

cumulative

vs.

non-cumulative

additive

vs.

multiplicative

and

per

string

vs.

per

time

step

noise

are

reported

in

this

paper

In

order

to

perform

many

experiments

in

a

reasonable

amount

of

computation

time

we

attempt

to

learn

the

simple

dual

parity

grammar

from

sample

strings

encoded

as

temporal

sequences

Dual

parity

is

a

automaton

that

recognizes

binary

strings

containing

an

even

number

of

ones

and

zeroes

-LRB-

see

figure

-RRB-

and

is

a

classic

example

of

a

hidden-state

problem

Both

sets

of

experiments

were

performed

on

the

dual

parity

problem

one

set

using

recurrent

networks

with

state

neurons

and

the

other

using

recurrent

networks

with

state

neurons

In

addition

another

set

of

simulations

apply

the

best

predicted

noise

model

-LRB-

additive

non-cumulative

per

Figure

The

Dual

Parity

Finite

State

Automaton

State

is

both

start

and

final

state

time

step

-RRB-

to

learning

a

slightly

larger

randomly

generated

state

automaton

-LRB-

-LCB-

see

figure

-RRB-

Our

experiments

consist

of

simulations

for

each

data

point

and

achieve

useful

-LRB-

-RRB-

confidence

levels

Second

order

recurrent

networks

with

input

neurons

are

trained

on

the

dual

parity

problem

The

finite

state

automaton

-LRB-

FSA

-RRB-

that

recognizes

the

dual

parity

grammar

is

shown

in

figure

The

data

consists

of

strings

of

lengths

to

The

training

set

consists

of

the

first

strings

of

lengths

to

while

the

initial

working

set

consists

of

strings

of

lengths

to

Generalization

performance

of

a

trained

network

was

obtained

by

testing

on

the

entire

data

set

of

strings

Training

was

stopped

when

all

of

the

training

set

was

correctly

classified

to

the

previously

discussed

error

criterion

If

training

persisted

beyond

epochs

training

was

halted

and

the

trained

network

was

not

used

Thus

only

networks

which

were

perfectly

trained

on

the

training

set

were

tested

for

generalization

For

runs

where

the

noise

amplitude

was

small

the

networks

usually

converged

in

less

than

epochs

For

higher

noise

values

up

to

a

quarter

of

the

runs

did

not

correctly

classify

the

training

set

in

the

required

number

of

epochs

and

were

discarded

Second

order

recurrent

networks

with

input

neurons

and

state

neurons

were

trained

on

the

grammar

The

finite

state

automaton

that

recognizes

this

grammar

is

shown

in

figure

The

data

consists

of

strings

of

length

to

The

training

set

consists

of

the

first

strings

If

training

persisted

beyond

epochs

training

was

halted

and

the

trained

network

was

not

used

As

mentioned

before

the

noise

source

is

chosen

to

be

a

random

zero-mean

white

noise

process

uniformly

distributed

between

-LRB-

a

a

-RRB-

where

a

is

the

amplitude

To

ensure

consistency

and

fairness

across

all

experiments

the

learning

rate

and

momentum

were

set

to

and

the

weights

were

initialized

to

random

values

between

-LSB-

-RSB-

in

all

simulations

The

empirical

results

presented

below

represent

confidence

levels

using

a

Student

's

t-distribution

and

were

obtained

by

running

simulations

for

each

data

point

The

convergence

and

generalization

results

for

the

dual

parity

problem

are

presented

first

followed

by

Figure

The

Randomly

Generated

State

Automaton

Double

circles

are

the

accepting

states

the

results

for

Generalization

Performance

In

figures

and

we

see

generalization

performance

on

the

dual

parity

problem

as

a

function

of

noise

amplitude

for

both

the

and

neuron

recurrent

nets

For

each

net

we

see

generalization

for

various

noise

combinations

For

all

simulations

the

zero-noise

case

is

where

a

Generalization

performance

mirrors

some

of

our

predictions

However

all

curves

exhibit

the

same

general

shape

of

generalization

performance

The

generalization

improves

as

noise

increases

from

the

zero-noise

point

for

all

noise

cases

and

then

starts

to

degrade

after

a

certain

noise-threshold

amplitude

After

this

noise-threshold

amplitude

generalization

performance

eventually

becomes

worse

than

the

zero-noise

case

-LRB-

The

curve

in

Figure

does

increase

as

a

increases

but

is

not

shown

-RRB-

The

cumulative

per

string

noise

cases

for

both

additive

and

multiplicative

noise

failed

to

converge

for

all

runs

Convergence

Time

Convergence

time

on

the

dual

parity

problem

as

a

function

of

noise

amplitude

are

plotted

in

figures

and

As

mentioned

above

networks

injected

with

cumulative

per

string

noise

do

not

converge

For

most

noise

models

convergence

for

added

noise

initially

decreases

up

to

a

noise-threshold

and

then

starts

to

increase

and

finally

exceeds

that

for

the

no-noise

case

Similar

behavior

for

feedforward

networks

has

been

observed

-LSB-

-RSB-

This

is

more

apparent

for

the

neuron

simulations

than

the

neuron

ones

however

close

inspection

will

show

there

is

a

convergence

mean

less

than

that

for

the

zero-noise

case

for

the

neuron

case

Training

Noise

Max

Amp

Mean

Generalization

Error

a

Training

Noise

Max

Amp

Mean

Generalization

Error

b

Training

Noise

Max

Amp

Mean

Generalization

Error

c

Training

Noise

Max

Amp

Mean

Generalization

Error

d

Training

Noise

Max

Amp

Mean

Generalization

Error

e

Training

Noise

Max

Amp

Mean

Generalization

Error

f

Figure

Mean

generalization

percent

error

as

a

function

of

noise

amplitude

a

for

recurrent

networks

with

neurons

The

error

bars

show

confidence

intervals

obtained

by

running

simulations

for

each

point

All

curves

in

the

top

row

are

for

additive

noise

and

all

in

the

bottom

row

for

multiplicative

noise

The

generalization

curves

are

for

-LRB-

a

-RRB-

non-cumulative

additive

per

time

step

-LRB-

b

-RRB-

non-cumulative

additive

per

string

-LRB-

c

-RRB-

cumulative

additive

per

time

step

-LRB-

d

-RRB-

non-cumulative

multiplicative

per

time

step

-LRB-

e

-RRB-

non-cumulative

multiplicative

per

string

-LRB-

f

-RRB-

cumulative

multiplicative

per

time

step

Experiments

with

cumulative

per

string

noise

do

not

converge

Training

Noise

Max

Amp

Mean

Generalization

Error

a

Training

Noise

Max

Amp

Mean

Generalization

Error

b

Training

Noise

Max

Amp

Mean

Generalization

Error

c

Training

Noise

Max

Amp

Mean

Generalization

Error

d

Training

Noise

Max

Amp

Mean

Generalization

Error

e

Training

Noise

Max

Amp

Mean

Generalization

Error

f

Figure

Mean

generalization

percent

error

as

a

function

of

error

magnitude

a

for

recurrent

networks

with

neurons

The

error

bars

show

confidence

intervals

obtained

by

running

simulations

for

each

data

point

All

curves

in

the

top

row

are

for

additive

noise

and

all

in

the

bottom

row

for

multiplicative

noise

The

generalization

curves

are

for

-LRB-

a

-RRB-

non-cumulative

additive

per

time

step

-LRB-

b

-RRB-

non-cumulative

additive

per

string

-LRB-

c

-RRB-

cumulative

additive

per

time

step

-LRB-

d

-RRB-

non-cumulative

multiplicative

per

time

step

-LRB-

e

-RRB-

non-cumulative

multiplicative

per

string

-LRB-

f

-RRB-

cumulative

multiplicative

per

time

step

Experiments

with

cumulative

per

string

noise

do

not

converge

Training

noise

max

amp

Mean

convergence

in

epochs

a

Training

noise

max

amp

Mean

convergence

in

epochs

b

Training

noise

max

amp

Mean

convergence

in

epochs

c

Training

noise

max

amp

Mean

convergence

in

epochs

d

Training

noise

max

amp

Mean

convergence

in

epochs

e

Training

noise

max

amp

Mean

convergence

in

epochs

f

Figure

Mean

convergence

in

epochs

as

a

function

of

noise

amplitude

a

for

recurrent

networks

with

state

neurons

The

error

bars

show

confidence

intervals

obtained

by

running

simulations

for

each

data

point

The

convergence

curves

are

for

-LRB-

a

-RRB-

non-cumulative

additive

per

time

step

-LRB-

b

-RRB-

non-cumulative

additive

per

string

-LRB-

c

-RRB-

cumulative

additive

per

time

step

-LRB-

d

-RRB-

non-cumulative

multiplicative

per

time

step

-LRB-

e

-RRB-

non-cumulative

multiplicative

per

string

-LRB-

f

-RRB-

cumulative

multiplicative

per

time

step

Experiments

with

cumulative

per

string

noise

do

not

converge

Training

noise

max

amp

Mean

convergence

in

epochs

b

Training

noise

max

amp

Mean

convergence

in

epochs

a

Training

noise

max

amp

Mean

convergence

in

epochs

c

Training

noise

max

amp

Mean

convergence

in

epochs

d

Training

noise

max

amp

e

Mean

convergence

in

epochs

Training

noise

max

amp

Mean

convergence

in

epochs

f

Figure

Mean

convergence

in

epochs

as

a

function

of

noise

amplitude

a

for

recurrent

networks

with

state

neurons

The

error

bars

show

confidence

intervals

obtained

by

running

simulations

for

each

data

point

The

convergence

curves

are

for

-LRB-

a

-RRB-

non-cumulative

additive

per

time

step

-LRB-

b

-RRB-

non-cumulative

additive

per

string

-LRB-

c

-RRB-

cumulative

additive

per

time

step

-LRB-

d

-RRB-

non-cumulative

multiplicative

per

time

step

-LRB-

e

-RRB-

non-cumulative

multiplicative

per

string

-LRB-

f

-RRB-

cumulative

multiplicative

per

time

step

Experiments

with

cumulative

per

string

noise

do

not

converge

Training

Noise

Max

Amp

Generalization

Error

a

b

Training

Noise

Max

Amp

Convergence

in

Epochs

a

b

Figure

Best

Convergence/Generalization

for

Additive

and

Multiplicative

Noises

-LRB-

a

-RRB-

multiplicative

noncumulative

per

time

step

-LRB-

b

-RRB-

additive

cumulative

per

time

step

Training

Noise

Max

Amp

Generalization

Error

II

a

b

Training

Noise

Max

Amp

Generalization

Error

I

a

b

Figure

-LRB-

I

-RRB-

Best

Generalization

for

Cumulative

and

Non-Cumulative

Noises

a

-RRB-

cumulative

additive

per

time

step

b

-RRB-

non-cumulative

additive

per

time

step

-LRB-

II

-RRB-

Best

Generalization

for

non-cumulative

Per

Time

Step

and

Per

String

Noises

a

-RRB-

per

string

additive

b

-RRB-

per

time

step

additive

Effects

of

each

Noise

Parameter

on

Performance

This

section

makes

use

of

the

dual

parity

simulations

to

assess

the

effects

of

each

noise

parameter

on

convergence

time

and

generalization

performance

In

order

to

make

fair

assessments

it

is

reasonable

to

compare

the

best

mean

performance

due

to

each

noise

parameter

For

example

to

compare

the

effects

of

additive

versus

multiplicative

noise

on

generalization

we

should

compare

the

mean

generalization

curves

of

networks

injected

with

additive

cumulative

per

time

step

noise

with

the

mean

generalization

curves

of

networks

injected

with

multiplicative

non-cumulative

per

time

step

noise

These

noises

represent

the

additive

and

multiplicative

noise

combinations

yielding

the

best

generalization

By

looking

at

the

best

noise

combinations

for

each

parameter

comparisons

can

be

made

which

are

independent

of

other

noise

issues

Using

the

above

case

as

an

example

if

the

best

additive

noise

show

better

generalization

than

the

best

multiplicative

noise

then

this

is

a

fair

indication

that

additive

noise

yields

better

generalization

than

multiplicative

noise

It

can

not

be

argued

that

perhaps

cumulative

noise

allows

better

generalization

since

if

this

was

the

case

then

we

would

be

comparing

additive

cumulative

per

time

step

noise

with

multiplicative

cumulative

per

time

step

noise

-LCB-

which

would

be

fair

if

such

was

the

case

Simulated

performance

closely

mirror

our

predictions

Improvements

were

observed

for

all

noise

models

except

for

cumulative

per

string

noises

which

failed

to

converge

for

all

runs

Generalization

improvement

was

more

emphasized

on

networks

with

states

while

convergence

enhancement

was

more

noticeable

on

networks

The

simulations

show

the

following

results

ffl

Additive

noise

is

better

tolerated

than

multiplicative

noise

achieves

better

generalization

and

yields

slightly

better

convergence

-LRB-

Figure

-RRB-

ffl

Cumulative

noise

achieves

optimal

generalization

and

convergence

at

lower

amplitudes

than

noncumulative

noise

Cumulative

noise

also

has

a

narrower

range

of

beneficial

noise

which

is

defined

as

the

range

of

noise

amplitudes

which

yields

better

performance

than

that

of

a

noiseless

network

-LRB-

Figure

illustrates

this

for

generalization

-RRB-

ffl

All

non-cumulative

per

time

step

noises

result

in

better

convergence

than

non-cumulative

per

string

noises

on

-LCB-

state

networks

Non-cumulative

additive

per

time

step

noise

yields

better

generalization

and

is

better

tolerated

than

non-cumulative

additive

per

string

noise

-LRB-

Figure

-RRB-

Overall

the

best

performance

is

obtained

by

applying

cumulative

and

non-cumulative

additive

noise

at

each

time

step

These

results

closely

match

the

predictions

of

section

The

only

exceptions

are

that

all

multiplicative

noise

models

seem

to

yield

equivalent

performance

This

discrepancy

between

prediction

and

simulation

may

be

due

to

the

detrimental

effects

of

weight

decay

in

multiplicative

noise

which

can

conflict

with

the

advantages

of

cumulative

and

per

time

step

noise

Generalization

vs.

Convergence

In

most

applications

both

generalization

and

convergence

are

important

improving

one

at

the

expense

of

the

other

becomes

a

difficult

decision

In

this

section

we

plot

the

trade-off

between

generalization

and

convergence

in

the

dual

parity

results

and

show

that

in

many

cases

it

is

possible

to

improve

both

performance

measures

simultaneously

In

figure

we

plot

generalization

vs.

convergence

Noise

magnitude

increases

from

left

to

right

in

each

curve

For

example

if

convergence

is

not

an

issue

than

one

would

pick

the

curve

that

dips

the

lowest

and

choose

the

corresponding

noise

amplitude

for

the

noise

method

corresponding

to

that

curve

These

plots

clearly

illustrate

that

additive

noise

methods

allow

the

best

generalization

and

convergence

for

the

neurons

networks

100 200 300 400

Mean

Convergence

in

Epochs

Mean

Generalzation

Error

a

100 200 300 400

Mean

Convergence

in

Epochs

Mean

Generalzation

Error

b

100 200 300 400

Mean

Convergence

in

Epochs

Mean

Generalzation

Error

c

100 200 300 400

Mean

Convergence

in

Epochs

Mean

Generalzation

Error

d

Figure

Generalization

percent

error

rate

vs.

convergence

in

epochs

The

curves

are

obtained

by

using

the

data

in

previous

figures

-LRB-

recall

simulations

for

each

data

point

-RRB-

The

lines

represent

the

following

-LCB-

non-cumulative

per

time

step

non-cumulative

per

string

cumulative

per

time

step

The

plots

represent

the

following

-LRB-

a

-RRB-

additive

noise

on

network

-LRB-

b

-RRB-

multiplicative

noise

on

network

-LRB-

c

-RRB-

additive

noise

on

network

-LRB-

d

-RRB-

multiplicative

noise

on

network

Noise

magnitude

increases

from

left

to

right

in

each

curve

Training

Noise

Max

Amplitude

Mean

Generalization

Error

a

Training

Noise

Max

Amplitude

Mean

Convergence

in

Epochs

b

Figure

Generalization

and

convergence

results

on

the

grammar

Additive

non-cumulative

per

time

step

noise

is

inserted

into

second

order

recurrent

networks

with

state

neurons

-LRB-

a

-RRB-

Mean

generalization

percent

error

as

a

function

of

noise

amplitude

-LRB-

b

-RRB-

Mean

convergence

in

epochs

as

a

function

of

noise

amplitude

The

error

bars

show

confidence

intervals

obtained

by

running

simulations

for

each

point

More

importantly

these

plots

illustrate

the

cases

where

both

convergence

and

generalization

are

improved

In

figures

and

the

curves

clearly

curl

down

and

to

the

left

for

lower

noise

amplitudes

before

rising

to

the

right

at

higher

noise

amplitudes

These

lower

regions

are

optimal

because

they

represent

improved

generalization

and

improved

convergence

i.e.

by

picking

a

noise

amplitude

from

this

hook

convergence

and

generalization

do

not

trade-off

Convergence

and

Generalization

Results

on

the

Grammar

Figures

and

plot

convergence

and

generalization

results

on

the

grammar

using

additive

noncumulative

per

time

step

noise

These

results

are

obtained

from

a

problem

slightly

more

difficult

than

the

dual

parity

problem

in

the

sense

that

the

grammar

is

slightly

larger

and

no

attempts

have

been

made

to

optimize

the

network

size

for

this

grammar

A

network

size

of

state

neurons

was

chosen

arbitrarily

although

more

neurons

could

have

been

added

to

improve

generalization

performance

We

simulated

only

the

best

predicted

noise

model

for

this

set

of

experiments

The

purpose

was

to

observe

the

performance

on

a

slightly

more

difficult

problem

As

shown

in

Figures

and

the

results

are

consistent

with

those

on

the

dual

parity

grammar

presented

above

Both

generalization

and

convergence

improved

for

a

range

of

noise

amplitudes

200 250 300 350

400 450 500 550

a

b

Mean

Convergence

in

Epochs

Mean

Generalization

Error

Figure

Generalization

percent

error

vs.

convergence

in

epochs

for

the

grammar

Additive

noncumulative

per

time

step

noise

is

inserted

into

second

order

recurrent

networks

with

state

neurons

Maximum

noise

magnitude

m

increases

from

point

a

-LRB-

m

-RRB-

to

point

b

-LRB-

m

-RRB-

CONCLUSIONS

We

have

presented

several

methods

of

injecting

synaptic

noise

to

recurrent

neural

networks

We

give

a

thorough

analysis

of

these

methods

and

empirically

test

these

methods

on

the

learning

dual

parity

and

grammars

from

strings

encoded

as

temporal

sequences

Results

show

that

most

of

these

methods

offer

several

benefits

on

recurrent

networks

ffl

Ability

to

improve

generalization

performance

ffl

Ability

to

improve

convergence

performance

ffl

Ability

to

improve

generalization

and

convergence

simultaneously

These

benefits

should

apply

to

feedforward

and

other

recurrent

networks

The

ability

to

improve

generalization

and

convergence

simultaneously

makes

these

results

novel

to

other

methods

previously

discussed

for

feedforward

nets

which

in

general

cast

convergence

as

a

cost

for

improved

generalization

performance

These

results

also

offer

some

insight

into

what

types

of

synaptic

noises

are

better

tolerated

when

training

under

noisy

environments

If

only

synaptic

noise

is

applicable

and

noise-free

updates

are

allowed

then

cumulative

multiplicative

noise

is

the

most

detrimental

at

high

amplitudes

In

addition

noise

which

is

uncorrelated

in

time

is

less

destructive

as

illustrated

by

the

fact

that

injecting

a

different

noise

at

each

time

step

-LRB-

for

one

input

string

-RRB-

allows

higher

detrimental

noise

thresholds

than

injecting

the

same

noise

at

each

time

step

-LRB-

effectively

equivalent

to

the

per

string

noise

models

-RRB-

However

the

appropriateness

of

any

of

these

noise

environments

needs

to

be

determined

It

would

be

interesting

to

apply

these

methods

to

other

problems

to

study

the

effect

of

problem

complexity

on

the

range

of

beneficial

noise

Further

research

with

larger

networks

could

determine

the

effect

of

network

size

on

noise

tolerance

and

effectiveness

on

generalization

and

convergence

Another

important

issue

is

the

relationship

between

the

range

of

optimal

noise

values

to

network

size

and

problem

complexity

Thus

it

could

be

possible

to

determine

the

usefulness

of

synaptic

noise

by

allowing

an

optimal

range

of

noise

values

to

be

predicted

for

each

application

ACKNOWLEDGEMENTS

We

would

like

to

thank

and

acknowledge

the

reviewers

for

valuable

comments

and

suggestions

and

acknowledge

Li

Chun

An

for

help

in

the

simulations

and

useful

discussions

APPENDIX

A

Series

Expansions

of

the

Error

Functions

The

Taylor

series

expansion

-LSB-

-RSB-

terms

of

the

error

functions

of

all

noise

models

are

derived

in

this

section

In

all

derivations

below

the

noise

is

assumed

to

be

zero-mean

white

and

uncorrelated

in

time

A.

Non-Cumulative

Noise

The

additive

per

time

step

noise

models

are

analyzed

in

detail

below

followed

by

straightforward

extensions

to

other

non-cumulative

noise

cases

A.

Additive

Per

Time

Step

Each

weight

Wt

ijk

is

perturbed

at

each

time

step

such

that

Wt

ijk

W

ij

k

t

ijk

-LRB-

-RRB-

where

fW

ij

kg

is

the

noise-free

weight

set

Performing

a

Taylor

series

expansion

on

the

output

of

state

neuron

STO

to

second

order

around

the

noise-free

weight

set

yields

STO

STO

T

X

t

X

ijk

t

ijk

@STO

@Wt

ijk

T

X

t

u

X

ijk

lmn

t

ijk?u

lmn

@Wt

ijk@Wu;lmn

O

-LRB-

-RRB-

-LRB-

-RRB-

The

error

Ep

for

pattern

p

on

a

noiseless

network

is

defined

as

Ep

-LRB-

-RRB-

where

fflp

is

fflp

STO

dp

-LRB-

-RRB-

However

the

injection

of

noise

represented

by

equation

-LRB-

-RRB-

augments

Ep

as

follows

Ep

-LSB-

STO

T

X

t

X

ijk

t

ijk

@STO

@Wt

ijk

T

X

t

u

X

ijk

lmn

t

ijk?u

lmn

@Wt

ijk@Wu;lmn

O

-LRB-

-RRB-

dp

-RSB-

-LRB-

-RRB-

Dropping

and

higher

order

terms

Ep

T

X

t

u

X

ijk

lmn

t

ijk?u

lmn

@STO

@Wt

ijk

@STO

@Wu

lmn

T

X

t

X

ijk

t

ijk

@STO

@Wt

ijk

fflp

T

X

t

u

X

ijk

lmn

t

ijk?u

lmn

@Wt

ijk@Wu;lmn

-LRB-

-RRB-

Since

the

noise

is

zero-mean

white

and

uncorrelated

taking

the

time

average

over

the

training

phase

for

the

per

time

step

case

yields

t

ijk

-LRB-

-RRB-

t

ijk?u

lmn

ffitijk

ulmn

-LRB-

-RRB-

where

ffitijk

ulmn

is

the

Kronecker

delta

Using

equations

-LRB-

-RRB-

-LRB-

-RRB-

simplifies

-LRB-

-RRB-

to

Enoisy

p

Ep

T

X

t

X

ijk

@STO

@Wt

ijk

fflp

-LRB-

Wt

ijk

-RRB-

-LRB-

-RRB-

It

is

important

to

realize

that

although

each

layer

has

the

same

set

of

weights

ijk

ijk

WT

ijk

-LRB-

-RRB-

nevertheless

their

respective

partial

derivatives

are

not

equal

because

they

are

time

dependent

@STO

ijk

@STO

ijk

@STO

@WT

ijk

-LRB-

-RRB-

Therefore

all

partial

derivatives

PT

t

P

ijk

-LSB-

-LRB-

@STO

@Wt

ijk

-RRB-

-LRB-

-LRB-

Wt

ijk

-RRB-

-RRB-

-RSB-

need

to

be

accounted

for

in

equation

-LRB-

-RRB-

A.

Additive

Per

String

To

extend

the

above

derivation

to

the

per

string

case

note

that

t

ijk

ijk

for

all

time

steps

within

each

string

in

eq

-LRB-

-RRB-

Therefore

t

ijk

t

ijk?u

lmn

ffiijk

lmn

-LRB-

-RRB-

and

eq

-LRB-

-RRB-

becomes

Enoisy

p

Ep

T

X

t

u

X

ijk

-LRB-

-RRB-

@STO

@Wt

ijk

@STO

@Wu

ijk

fflp

@Wt

ijk@Wu;ijk

-LRB-

-RRB-

A.

Multiplicative

Per

Time

Step

and

Per

String

The

above

per

string

and

per

time

step

derivations

can

be

easily

extended

to

the

multiplicative

noise

models

by

substituting

Wt

ijk

W

ij

k

W

ij

k?t

ijk

-LRB-

-RRB-

in

place

of

eq

-LRB-

-RRB-

A.

Cumulative

Noise

The

cumulative

noise

models

are

analyzed

below

They

are

based

on

and

described

in

the

same

format

as

the

above

section

on

non-cumulative

noise

A.

Additive

Per

Time

Step

In

the

cumulative

additive

per

time

step

case

each

noise

injection

step

is

such

that

Wt

ijk

W

ij

k

t

X

o

o

ijk

-LRB-

-RRB-

The

time

averages

t

X

o

o

ijk

-LRB-

-RRB-

tX

o

o

ijk

u

X

lmn

ffiijk

lmn

-LRB-

-RRB-

where

v

min

-LRB-

t

u

-RRB-

can

be

used

to

simplify

the

error

expression

A.

Additive

Per

String

The

noise

injection

step

for

the

additive

per

string

case

can

be

modeled

by

Wt

ijk

W

ij

k

-LRB-

t

-RRB-

ijk

-LRB-

-RRB-

The

time

averages

-LRB-

t

-RRB-

ijk

-LRB-

-RRB-

-LRB-

t

-RRB-

ijk

-LRB-

u

-RRB-

lmn

-LRB-

t

-RRB-

-LRB-

u

-RRB-

ffiijk

lmn

-LRB-

-RRB-

can

be

used

to

simplify

the

error

expression

A.

Multiplicative

Per

Time

Step

For

the

multiplicative

per

time

step

case

each

noise

injection

step

is

such

that

Wt

ijk

W

ij

k

tY

o

-LRB-

o

ijk

-RRB-

-LRB-

-RRB-

To

put

eq

-LRB-

-RRB-

in

a

similar

format

for

the

error

expansion

note

that

it

can

be

represented

by

Wt

ijk

W

ij

k

W

ij

k?t

ijk

-LRB-

-RRB-

where

t

ijk

tY

o

-LRB-

o

ijk

-RRB-

-LRB-

-RRB-

The

time

averages

t

ijk

-LRB-

-RRB-

t

ijk?u

lmn

ffiijk

lmn

-LRB-

-RRB-

where

v

min

-LRB-

t

u

-RRB-

can

be

used

to

simplify

the

error

expression

A.

Multiplicative

Per

String

In

the

per

string

case

the

noise

injection

step

Wt

ijk

W

ij

k

-LRB-

ijk

-RRB-

t

can

be

represented

by

Wt

ijk

W

ij

k

W

ij

k?t

ijk

-LRB-

-RRB-

where

t

ijk

-LRB-

ijk

-RRB-

t

-LRB-

-RRB-

Using

binomial

expansion

and

dropping

third

or

higher

terms

we

get

the

following

approximations

t

ijk

-LRB-

t

-RRB-

ijk

t

-LRB-

t

-RRB-

O

-LRB-

-RRB-

-LRB-

t

-RRB-

-LRB-

-RRB-

t

ijk?u

lmn

-LRB-

ijk

-RRB-

t

-LRB-

lmn

-RRB-

u

-LRB-

ijk

-RRB-

t

-LRB-

lmn

-RRB-

u

-LRB-

ijk

-RRB-

t

-LRB-

lmn

-RRB-

u

t

-LRB-

t

-RRB-

-LRB-

u

-RRB-

t

-LRB-

t

-RRB-

u

-LRB-

u

-RRB-

-LRB-

t

-RRB-

-LRB-

u

-RRB-

ffiijk

lmn

-LRB-

t

-RRB-

-LRB-

u

-RRB-

-LRB-

t

-RRB-

-LRB-

u

-RRB-

ffiijk

lmn

-LRB-

-RRB-

For

all

cases

the

weight

update

Wijk

descends

the

noise-enhanced

error

function

Enoisy

p

B

Cumulative

vs.

Non-cumulative

Noise

in

Feedforward

Net

works

In

feedforward

networks

cumulative

and

non-cumulative

noise

models

are

identical

if

using

per

sample

training

but

can

have

different

variance

if

batch

training

is

used

-LRB-

see

analysis

below

-RRB-

The

methods

described

in

this

paper

update

the

set

of

noise-free

weights

and

accumulate

noise

between

weight

updates

as

opposed

to

the

continuous

accumulation

of

noise

across

weight

updates

Per

sample

updating

and

batch

updating

are

considered

below

B.

Per

Sample

Updating

Since

the

length

of

each

sample

-LRB-

in

time

-RRB-

is

one

noise

can

not

accumulate

in

the

weights

because

we

do

not

keep

track

of

the

previous

noise

values

after

each

weight

update

Thus

cumulative

and

non-cumulative

noise

models

are

identical

when

updating

the

weights

after

the

presentation

of

each

training

sample

B.

Batch

Training

In

this

case

the

errors

for

several

samples

are

computed

before

the

weights

are

updated

For

example

in

an

additive

noise

model

the

weights

are

augmented

as

follows

between

updates

for

each

training

sample

ti

and

weight

Wj

Wj

Wj

Wj

Wj

Wj

Wj

tM

Wj

Wj

nM

nM

M

where

M

is

the

number

of

training

samples

in

the

batch

i

are

uncorrelated

noise

values

and

ni

are

the

effective

non-cumulative

noise

values

Although

the

terms

ni

are

correlated

the

correlations

do

not

contribute

to

the

augmented

error

function

in

the

feedforward

case

-LRB-

They

do

contribute

in

recurrent

architectures

as

shown

in

Appendix

A

-RRB-

Thus

inserting

cumulative

noise

i

between

updates

is

effectively

equivalent

to

inserting

noncumulative

noise

values

ni

between

updates

The

noise

values

ni

are

also

zero-mean

however

the

variance

increases

linearly

with

the

number

of

training

samples

in

the

batch

This

is

easily

shown

by

using

the

following

equality

V

ar

-LRB-

aX

bY

-RRB-

ar

-LRB-

X

-RRB-

ar

-LRB-

Y

-RRB-

-LRB-

X

Y

-RRB-

-LRB-

-RRB-

where

a

and

b

are

constants

X

and

Y

are

random

variables

V

ar

-LRB-

-RRB-

is

the

variance

and

Cov

-LRB-

-RRB-

is

the

covariance

Applying

this

equality

here

X

and

Y

are

independent

noise

values

a

b

Cov

-LRB-

X

Y

-RRB-

and

the

following

is

obtained

V

ar

-LRB-

nM

-RRB-

V

ar

-LRB-

-RRB-

V

ar

-LRB-

-RRB-

V

ar

-LRB-

M

-RRB-

-LRB-

-RRB-

Thus

the

noise

values

for

training

samples

presented

later

within

the

batch

will

observe

greater

variance

References

-LSB-

-RSB-

D.H.

Ackley

G.E.

Hinton

and

T.J.

Sejnowski

A

learning

algorithm

for

boltzmann

machines

Cognitive

Science

-LCB-

-LSB-

-RSB-

E.B.

Baum

and

D.

Haussler

What

size

net

gives

valid

generalization

Neural

Computation

-LCB-

-LSB-

-RSB-

C.

M.

Bishop

Training

with

noise

is

equivalent

to

Tikhonov

regularization

Neural

Computation

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

H.

J.

Bremermann

and

R.

W.

Anderson

An

alternative

to

back-propagation

A

simple

rule

of

synaptic

modification

for

neural

net

training

and

memory

Technical

Report

PAM

-LCB-

U.

C.

Berkeley

Center

for

Pure

and

Applied

Mathematics

-LSB-

-RSB-

H.

J.

Bremermann

and

R.

W.

Anderson

How

the

brain

adjusts

synapses

maybe

In

R.

S.

Boyer

editor

Automated

Reasoning

Essays

in

Honor

of

Woody

Bledsoe

chapter

pages

-LCB-

Kluwer

Academic

Publishers

Boston

MA

-LSB-

-RSB-

Robert

M.

Burton

Jr.

and

George

J.

Mpitsos

Event-dependent

control

of

noise

enhances

learning

in

neural

networks

Neural

Networks

-LCB-

-LSB-

-RSB-

Gert

Cauwenberghs

A

fast

stochastic

error-descent

algorithm

for

supervised

learning

and

optimization

In

S.J

Hanson

J.D.

Cowan

and

C.L.

Giles

editors

Advances

in

Neural

Information

Processing

Systems

pages

-LCB-

San

Mateo

CA

Morgan

Kaufmann

Publishers

-LSB-

-RSB-

Yann

Le

Cun

John

S.

Denker

and

Sara

A.

Solla

Optimal

brain

damage

In

D.

Touretzky

editor

Advances

in

Neural

Information

Processing

Systems

pages

-LCB-

Morgan

Kaufmann

Publishers

-LSB-

-RSB-

A.

Dembo

and

T.

Kailath

Model-free

distributed

learning

IEEE

Transactions

on

Neural

Networks

-LCB-

-LSB-

-RSB-

Barry

Flower

and

Marwan

Jabri

Summed

weight

neuron

perturbation

An

O

-LRB-

n

-RRB-

improvement

over

weight

perturbation

In

S.J

Hanson

J.D.

Cowan

and

C.L.

Giles

editors

Advances

in

Neural

Information

Processing

Systems

pages

-LCB-

San

Mateo

CA

Morgan

Kaufmann

Publishers

-LSB-

-RSB-

C.L.

Giles

C.B.

Miller

D.

Chen

H.H.

Chen

G.Z.

Sun

and

Y.C.

Lee

Learning

and

extracting

finite

state

automata

with

second-order

recurrent

neural

networks

Neural

Computation

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

C.L.

Giles

and

C.W.

Omlin

Pruning

recurrent

neural

networks

for

improved

generalization

performance

IEEE

Transactions

on

Neural

Networks

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

Stephen

Jos?e

Hanson

A

stochastic

version

of

the

delta

rule

Physica

D.

-LCB-

-LSB-

-RSB-

G.E.

Hinton

and

T.J.

Sejnowski

Learning

and

relearning

in

Boltzmann

machines

In

Parallel

Distributed

Processing

chapter

pages

-LCB-

MIT

Press

Cambridge

MA

-LSB-

-RSB-

Geoffrey

E.

Hinton

Terrence

J.

Sejnowski

and

David

H.

Ackley

Boltzmann

machines

Constraint

satisfaction

networks

that

learn

Technical

Report

Carnegie

-LCB-

Mellon

University

CS

Dept.

May

-LSB-

-RSB-

Lasse

Holmstr?om

and

Petri

Koistinen

Using

additive

noise

in

back-propagation

training

IEEE

Transactions

on

Neural

Networks

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

Marwan

Jabri

and

Barry

Flower

Weight

perturbation

An

optimal

architecture

and

learning

technique

for

analog

vlsi

feedforward

and

recurrent

multilayer

networks

IEEE

Transactions

on

Neural

Networks

pages

-LCB-

-LSB-

-RSB-

Stephen

Judd

and

Paul

W.

Munro

Nets

with

unreliable

hidden

nodes

learn

error-correcting

codes

In

S.J

Hanson

J.D.

Cowan

and

C.L.

Giles

editors

Advances

in

Neural

Information

Processing

Systems

pages

-LCB-

San

Mateo

CA

Morgan

Kaufmann

Publishers

-LSB-

-RSB-

Danny

Kilis

and

Eugene

Ackerman

A

stochastic

neuron

model

for

pattern

recognition

In

Proceedings

of

the

International

Joint

Conference

on

Neural

Networks

volume

Washington

D.C.

January

-LSB-

-RSB-

S.

Kirkpatrick

Jr.

C.D.

Galett

and

M.P.

Vecchi

Optimization

by

simulated

annealing

Science

-LCB-

May

-LSB-

-RSB-

Anders

Krogh

and

John

A.

Hertz

A

simple

weight

decay

can

improve

generalization

In

J.E.

Moody

S.J.

Hanson

and

R.P.

Lippmann

editors

Advances

in

Neural

Information

Processing

Systems

pages

-LCB-

San

Mateo

CA

Morgan

Kaufmann

Publishers

-LSB-

-RSB-

C.B.

Miller

and

C.L.

Giles

Experimental

comparison

of

the

effect

of

order

in

recurrent

neural

networks

International

Journal

of

Pattern

Recognition

and

Artifical

Intelligence

-LRB-

-RRB-

-LCB-

Special

Issue

on

Neural

Networks

and

Pattern

Recognition

editors

I.

Guyon

P.S.P.

Wang

-LSB-

-RSB-

Jay

I.

Minnix

Fault

tolerance

of

the

backpropagation

neural

network

trained

on

noisy

inputs

In

Proc

of

IJCNN

volume

I

pages

I

-LCB-

-LCB-

-LSB-

-RSB-

Alan

F.

Murray

and

Peter

J.

Edwards

Enhanced

MLP

performance

and

fault

tolerance

resulting

from

synaptic

weight

noise

during

training

IEEE

Trans

on

Neural

Networks

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

C.W.

Omlin

and

C.L.

Giles

Fault-tolerant

implementation

of

finite-state

automata

in

recurrent

neural

networks

Technical

Report

Computer

Science

Department

Rensselaer

Polytechnic

Institute

Troy

NY

-LSB-

-RSB-

Russel

Reed

Pruning

algorithms

-LCB-

a

survey

IEEE

Trans

on

Neural

Networks

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

R.J.Williams

and

D.

Zipser

Gradient-based

learning

algorithms

for

recurrent

networks

and

their

computational

complexity

In

Y.

Chauvin

and

D.

E.

Rumelhart

editors

Back-propagation

Theory

Architectures

and

Applications

chapter

pages

-LCB-

Lawrence

Erlbaum

Publishers

Hillsdale

N.J.

-LSB-

-RSB-

Carlo

H.

S?equin

and

Reed

D.

Clay

Fault

tolerance

in

artificial

neural

networks

In

Proc

of

IJCNN

volume

I

pages

I

-LCB-

-LCB-

-LSB-

-RSB-

R.L.

Watrous

and

G.M.

Kuhn

Induction

of

finite-state

languages

using

second-order

recurrent

networks

Neural

Computation

-LRB-

-RRB-

-LSB-

-RSB-

Z.

Zeng

R.M.

Goodman

and

P.

Smyth

Learning

finite

state

machines

with

self-clustering

recurrent

networks

Neural

Computation

-LRB-

-RRB-

-LCB-

additive noise
automata
convergence
generalization
multiplicative noise
recurrent neural networks
regularization
synaptic noise
temporal sequences