Beneficial roles for chaotic
variability in learning systems

Abstract. Neural responses are highly variable, even under identical task conditions. Significant eﬀorts are being
directed toward explaining how the brain copes with and may even leverage such variability to help learn thetask and environment. Here we explore the issue in a recurrent neural network model that is trained to classify
inputs. We find two potential beneficial roles for chaotic variability in these systems: (1) chaos can accelerate theflexible relearning of a task after it is modified; and (2) chaos can lift the network representation of data into ahigher-dimensional space, which allows the network to classify inputs embedded in low-dimensional spaces.

Goal: investigate how the initial dynamics of a recurrent neural network(RNN) influence the solutions learned by the network when trained to do atask.
We characterize these dynamics by how chaotic the trajectories are.

RNN

inputsclass 1

InputsReadout

class 2

···

input

evaluationdelay

Train recurrent and output weights with stochastic gradient descent (SGD)
2

Goal: investigate how the initial dynamics of a recurrent neural network(RNN) influence the solutions learned by the network when trained to do atask.
We characterize these dynamics by how chaotic the trajectories are.
Less chaotic
initialization
Activations of ten
neural units throughtime, before training

More chaotic
initialization

RNN

inputsclass 1

InputsReadout

class 2···

input

evaluationdelay

Train recurrent and output weights with stochastic gradient descent (SGD)3

Let’s choose a task to train the network on
Train with SGD, using Categorical Cross Entropy loss

inputsclass 1

Delayedclassification
task···

input
inputs are gaussianballs randomly
distributed in spacedelay
The input is only
shown to the networkat time t = 0. After
this the network
evolves autonomously

4class 2The error between thenetwork output andthe class label is used
to update the weights.

evaluation (t=10)

The network state isread out and used toclassify the input attime t = 10

Let’s choose a task to train the network on
Train with SGD, using Categorical Cross Entropy loss

inputsclass 1

Delayedclassification
task

These inputs induce a
response in thenetwork. Hereh(x, t = 0) is the
response of thenetwork to input
vector x at time t = 0.

···

input

h4

h3

h2

delay

h1

recurrent unit activations
5class 2evaluation (t=10)

Let’s choose a task to train the network on
Train with SGD, using Categorical Cross Entropy loss

inputsclass 1···

inputThis response evolves
through time. This is
only a schematic
though. Let’s see
what happens when
we actually train thenetwork.

h4

h3

h2

delay

h1

recurrent unit activations
6class 2evaluation (t=10)

High-dimensional,linearly separable input dataclass 1class 2···evaluation (t=10)

delay

PC2input

PC17

High-dimensional,
linearly separable input dataclass 1class 2···

evaluation (t=10)

delay

PC2input

The network forms stable fixed points
PC1
7

t=0t = 10

t=5How do we characterize thegeometry of these?
Eﬀective dimensionality (ED) isone approach
Rajan et al., NeurIPS (2010)
before training
after training

40

ED

30
20
1000

510t
815

t=0t = 10

t=5How do we characterize thegeometry of these?
Eﬀective dimensionality (ED) isone approach
Rajan et al., NeurIPS (2010)

λ2

=λ1

DiagonalizationV

λ1
λ2VTd∑λ( k=1 k)

2ED =d∑k=1 λk2

before training
after training

40
30

EDCovariance

20
10
00

ED=2

ED=1

ED=1.3
8

510

t

15

High-dimensional,
linearly separable input data

What happens if we initialize the network tobe more chaotic?

class 1class 2···

evaluation (t=10)

delay

PC2

input

PC1
9

High-dimensional,
linearly separable input dataclass 1class 2···

input
input

delaydelay(t=10)
evaluation
evaluationt = 20The network forms low-dimensional chaotic attractors
10

If the labels are changed,
this representation may beproblematic due to tightoverlap between points
belonging to diﬀerent
classes

Both networks have perfect classification accuracy at the end of training.
What happens if the task is changed in the middle of training?

11

Switching task labels
Error regions are over 5
network and task
instantiations (see Methods)
Less chaotic
More chaotic

At epoch 10, 50% of the labels are switched

The more chaotic network is better at recovering after the label switch.

12

Switching cluster locations
Error regions are over 5
network and task
instantiations (see Methods)
Less chaotic
More chaotic

At epoch 10, the input clusters are moved to new random locations

The more chaotic network is better at recovering after the label switch.

13• Before we had high-dimensional, linearly separable input data.
• Now we’ll look at the case of very low-dimensional input dataclass 1class 2···

evaluation (t=10)

delayPC2

input

PC1
14

• Before we had high-dimensional, linearly separable input data.
• Now we’ll look at the case of very low-dimensional input data
2-dimension inputs

class 1class 2···

input

evaluation (t=10)

delay• In this case, it is useful to expand dimensionality to help separate
out the classes.
• Chaotic dynamics is one mechanism for this expansion(Legenstein and Maass, Neural Networks (2007)).

15

• Before we had high-dimensional, linearly separable input data.
• Now we’ll look at the case of very low-dimensional input data
2-dimension inputsclass 1class 2···

input

evaluation (t=10)

delay• In this case, it is useful to expand dimensionality to help separate
out the classes.
• Chaotic dynamics is one mechanism for this expansion(Legenstein and Maass, Neural Networks (2007)).

Chaotic divergence15• The more chaotic network expands

dimensionality before training as well as
after training.
• The less chaotic network is not able tolearn this expansion.

to do better at the task.

less chaotic
more chaotic

10

before training
after training

5005

t10

accuracy
testingaccuracy

ED15• This allows the more chaotic network

1.000.75
less chaotic

0.50

more chaotic

0.0 2.5 5.0 7.5 ×104

15

# trainingsamples

16

Conclusions• Chaotic dynamics can have utility in recurrent neural network models:• Leads to solutions that can more quickly adapt to changing data.
• Expands dimensionality, which can help when classifying low-dimensional data.
• This suggests beneficial attributes of the variability seen in biology, which may inpart be generated by chaotic dynamics.

17

Methods• Network equations

ht = tanh(Wht−1 + xt + b)ot̂ = Rht + b′

recurrent unit activations

readout• Chaos was modulated by increasing the variance of the connections between neurons atinitialization: W = (1 − ϵ)I + (gϵ/ N)J where ϵ = .01 and J is a matrix with standard
normal entries. For the less chaotic network g = 5, and for the more chaotic networkg = 250.
• Error regions on plots are found by fitting a gamma distribution to the data, and shading
the region containing 75% probability mass of the distribution. For the accuracy plots
this distribution was reflected around the y-axis and shifted by +1 (since accuracy isbounded above by 1), and for the dimensionality plots the distribution was shifted by +1(since the eﬀective dimensionality is bounded from below by 1).

18