Learning to memorize input--output
mapping as bifurcation in neural dynamics:
relevance of multiple timescales for synapsechanges
Tomoki Kurikawa & Kunihiko Kaneko

Neural Computing and Applications
ISSN 0941-0643
Volume 21
Number 4
Neural Comput & Applic (2012)
21:725-734
DOI 10.1007/s00521-011-0650-2

1 23

Your article is protected by copyright and allrights are held exclusively by Springer-Verlag
London Limited. This e-offprint is for personaluse only and shall not be self-archived inelectronic repositories. If you wish to selfarchive your work, please use the accepted
author’s version for posting to your own
website or your institution’s repository. You
may further deposit the accepted author’s
version on a funder’s repository at a funder’srequest, provided it is not made publicly
available until 12 months after publication.

1 23

Author's personal copy
Neural Comput & Applic (2012) 21:725–734
DOI 10.1007/s00521-011-0650-2

ICONIP2010

Learning to memorize input–output mapping as bifurcationin neural dynamics: relevance of multiple timescales
for synapse changes
Tomoki Kurikawa • Kunihiko Kaneko

Received: 27 April 2011 / Accepted: 7 May 2011 / Published online: 21 May 2011
Ó Springer-Verlag London Limited 2011

Abstract When a certain input–output mapping ismemorized, the neural dynamics provide a prescribedneural activity output that depends on the external input.
Without such an input, neural states do not provide memorized output. Only upon input, memory is recalled as an
attractor, while neural activity without an input need not
fall on such attractor but can fall on another attractor
distinct from the evoked one. With this background, we
propose that memory recall occurs as a bifurcation from thespontaneous attractor to the corresponding attractor
matching the requested target output, as the strength of theexternal input is increased. We introduce a neural networkmodel that enables the learning of such memories asbifurcations. After the learning process is complete, theneural dynamics are shaped to generate a prescribed target
in the presence of each input. We find that the capacity ofsuch memory depends on the timescales for the neuralactivity and synaptic plasticity. The maximal memory
capacity is achieved at a certain relationship between thetimescales, where the residence time at previous learned
targets during the learning process is minimized.
keywords Neural network  Bifurcation 
Associative reward–penalty  Multiple timescales

1 Introduction
The ability to learn appropriate responses to stimuli fromthe environment is one of the most important features ofT. Kurikawa (&)  K. Kaneko
Department of Basic Science, University of Tokyo,
Komaba 3-8-1, Meguro-ku, Tokyo, Japane-mail: kurikawa@complex.c.u-tokyo.ac.jp

the brain. On the basis of the advances made in neuroscience over the past decades, it is now generally acceptedthat the output responses to input stimuli are memorized by
the synaptic plasticity. Various neural network models
have been proposed to study how a synaptic strength pattern is formed to memorize given input–output (I/O)
mappings. In most of these studies [1–5], inputs are supplied as the initial states for neural activity, whose temporal
evolution results in the generation of desired outputs. Thus,a dynamical system without inputs is not taken into
account, and spontaneous activity is only considered to benoise. However, recent experimental studies have shownthat structured spontaneous neural activity exists even in
the absence of external stimuli and that such spontaneous
activity is related to the cognitive function [6–9].
Upon external stimuli, such neural activity is modified
to provide appropriate response neural activities [10].
Considering these experiments, we proposed a novel perspective for the memorization of I/O mappings [11]. If aneural system memorizes an I/O mapping, the spontaneous
dynamics are modulated by a memorized input to provide a
required target output. In other words, the input changes
the flow structure of the neural dynamical system to generate the corresponding target output. This change in flow
is mathematically formulated as the bifurcations of
attractor upon input.
In the present paper, we use a layered neural networkmodel to show that this idea of ‘‘memories as bifurcations’’is indeed realized. In particular, we discuss the dependenceof the learning process on the neural and synaptic timescales: the timescales for neural activity and the plasticities
of both forward and backward synapses. We find that there
is an optimal relation among these timescales, in order forthe system to have a maximum memory capacity. This
relation states that the timescale for neural activity is123Author's personal copy
726

smaller than that for the backward synaptic plasticity andthat the timescale for the backward synaptic plasticity issmaller than that for the forward one. We discuss how thistimescale relationship is relevant to the formation of flowof neural dynamical system to achieve appropriate bifurcations for memories.
The structure of this paper is as follows: In the second
section, we describe the model used in the paper. We adopt
reinforcement learning in a three-layered network, which iscomposed of two rules, Hebbian (strengthening with‘‘reward’’) and anti-Hebbian (weakening with ‘‘punishment’’) synaptic changes. We describe the dynamics ofneural activity and the synaptic dynamics, i.e., the change
in synaptic strength by reinforcement learning. The next
section shows the results of the model described in thepreceding section; these results include the typical behaviorof our model and a demonstration of ‘‘memories as bifurcations.’’ In particular, how the neural and synaptic
dynamics change depending on the relationship between
their timescales is analyzed in depth. In the last section, we
discuss the implications of the results for learning andmemory in our brain and the significance of the multiple
timescales.

Neural Comput & Applic (2012) 21:725–734

error signal, which is given as the distance between theactivity pattern of the output neurons and a prescribedtarget output pattern. During the learning process, thesynaptic strength is changed in accordance with the Hebbian and anti-Hebbian rules, switching based on the magnitude of the error signal (Fig. 1).
To be specific, we adopt the following model with
N neurons in each layer. Three types of synapses are
considered: forward synapses (FSs), backward synapses(BSs), and mutually inhibiting intralayer synapses (ISs).
FSs connect the neurons in the input layer to those in thehidden layer and the neurons in the hidden layer to those in
the output layer. BSs connect the neurons in the output
layer to those in the hidden layer, while ISs connect theneurons within a given layer (a hidden layer or output
layer).
2.1 Neural dynamics
The neural activity in the input layer is determined by an
input pattern, I, a vector whose element takes the value 0 or
1; the magnitude of this vector (input strength) is g (1). The
neural activities in the other layers change, as shown by therate coding model (2, 3):

2 Model

123

1 0

0

0 1

0

0 0

1

er
ay

l
ut

p

In

en

er
lay

Error signal

We construct a neural network model for learning basedon the following two conditions that satisfy the biological requirements for the normal process of the brain
function. (1) Different error information for each individual neuron should not be required. In other words,
individual error information is used commonly for all
neurons. In contrast, in the error back-propagation algorithm [5], one of the most popular learning algorithms
for neural networks, information corresponding to eachof the output neurons is required. In biological learningwith a neural system, however, it is difficult to transmit
the specified information to each neuron. (2) I/O mappings should be learned sequentially, one-by-one, i.e., a
new I/O mapping should be learned only after the previous mapping has been learned, while preserving thepreviously learned mappings. In contrast, in most learning algorithms for neural networks, numerous mappings
are simultaneously and iteratively learned by graduallychanging the synaptic strength until all of the mappings
are memorized.
To satisfy the above-mentioned requirements, we
introduce a layered network model consisting of input,
hidden, and output layers, along with a reinforcement
learning algorithm known as the associative reward–penalty (ARP) algorithm (Fig. 1) [4, 12]. In this model, several
I/O mappings are learned one-by-one using only a single

dd

Hi

er
ay

l
ut

p

ut

O

t

ge

-

r
Ta

FS

BS

IS

Fig. 1 Schematic representation of network architecture of ourmodel. In this study, we use a three-layered model with input,
hidden, and output layers. There are three kinds of synaptic
connections in the network: forward synapses (FSs) represented withthe blue lines, backward synapses (BSs) with the red lines, andinhibitory synapses (ISs) with the gray lines. FSs and BSs are
modified during learning in accord with the error signal (the greenline) between the output activity and target output activity, while ISs
are fixed

Author's personal copy
Neural Comput & Applic (2012) 21:725–734

xi ¼ gIi

ðI 2 f0; 1gÞðinput layerÞ

sNA x_i ¼ 1=ð1 þ expðbuhidi þ hÞÞ  xi

727

ð1Þ
ðhidden layersÞ
ð2Þ

sNA x_i ¼ 1=ð1 þ expðbuouti þ hÞÞ  xiðoutput layersÞ
ð3Þ

where xi is the firing rate of neuron i, and ui is the input
current applied to neuron i. The input current is given byP
PN BS out P
inuhid= Nj=1 JFS? j=i JIS xhidforiij xj ?
j=1 Jij xjjPNoutthe neurons in the hidden layer and ui = j=1 JFS
ijP
IS outhid
xj ? j=i J xj for the neurons in the output layer.
BS
Here, JFS
ij (Jij ) is the strength of the forward (backward)
synapse from a pre-synaptic neuron, j, to a post-synaptic
neuron, i. JIS is the strength parameter for the mutuallyinhibiting IS; this parameter assumes a fixed and identicalvalue for all synapses except JIS
ii = 0 for all i. The
parameters are set at sNA = 1, b = 43, h = 2.5, g = 1.0,
JIS = - 1.0, and N = 10.
For each input pattern, we prescribe a target output pattern, n, as an N-dimensional vector whose element takes thevalue 0 or 1. Sparse input and target patterns, in which only
one neuron is activated, are chosen. By describing the neuralactivity in the output layer as the N-dimensional vector
Xout, the learning task moves the error, E = |Xout - n|2/N,
closer to zero, as measured by the Euclidean norm.

Here, r is the spontaneous firing rate (set at 0.1), and e isthe error threshold, set at 10-3. The sign of R changes withthe magnitude of the error signal, E, between the output
pattern and target pattern. When the output pattern is close
to the target pattern, i.e., E ¼ jX out  nj2 =N  e; thesynaptic plasticity follows the Hebbian rule, which isderived by substituting RFS = 1 and RBS = 0 in (4). This
plasticity stabilizes the ongoing neural activity pattern.
Note that during this stabilization process, only the strengthof the FS varies, and hence, memories of the I/O mappings
are embedded in the FSs. In contrast, when the output
pattern is distant from the target pattern, i.e., E [ e; thesynaptic plasticity follows the so-called anti-Hebbian rule,and the ongoing neural activity pattern is destabilized. Notethat with the above form, (xi - r)xj, the synapse shows
only negligible changes when its pre-synaptic neuron, j, isin a low-firing state. In our model, we require only a single
error term for all of the neurons; this is in strong contrast toerror back propagation, which requires the computation ofa large number of error terms, i.e., as many error terms as
output neurons.
In most neural network studies, only two timescales are
considered: one for neural activities and the other forsynaptic plasticity. In this study, we consider a variety oftimescales for synaptic plasticity and introduce two timescales for the plasticities of the FSs and BSs. As will beshown later, I/O mappings are successfully memorized
when the difference between the timescales is appropriate.

2.2 Synaptic dynamics
Synaptic plasticity is necessary for learning in neural network models. We adopt two schemes for the synaptic
plasticity: multiple timescales and the ARP algorithm forreinforcement learning [4, 12]. First, the timescale of theplasticity of FSs sFS is different from that of the plasticityof BSs sBS. Second, the synapse pattern that generates thetarget output pattern is strengthened, in accordance with
the Hebbian rule; A connection between the active neurons
is strengthened, and, otherwise, it is unchanged or
depressed. On the other hand, the synapse pattern that does
not generate the target output pattern is weakened as per
the anti-Hebbian rule; A connection between the active
neurons is depressed, and, otherwise, it is unchanged orstrengthened. In accordance with the ARP algorithm, thesynaptic dynamics depend on the activities of the pre- andpost-synaptic neurons as well as on the R determined fromerror signal E:
sp J_ijp ¼ Rp ðxi  rÞxjðJ  0Þðp ¼ FS or BSÞ ð4Þ
1 for E  e
0for E  e
RBS ¼
RFS ¼
1 for E [ e1 for E [ eð5Þ

3 Results
3.1 Learning process
We first show that this model can be used to learn I/O
mappings. Our learning process is composed of two phases.
The neural activity varies to search a desired target when
far from it (E [ e) and stabilizes when matching it (E\e).
Figure 2 shows the learning process for two I/O mappings.
When the error is large, the present neural activity becomes
unstable by the anti-Hebbian rule (5), and hence, the neuralactivity itinerates among different patterns (0 \ t \ 350).
It searches the target pattern during this itineration. We
term this period the ‘‘search’’ phase in what follows. At
t*350, the magnitude of the error decreases to a sufficient
extent when the output dynamics are within the neighborhood of e: Once this occurs, the neural activity is stabilized
as per the Hebbian rule (Eq. 5), and the output activity
remains close to the target (350 \ t \ 800). In this model,
only one neuron in the output layer is active in each of thetarget patterns (the neurons highlighted by the blue andgreen bars in Fig. 2a), along with one neuron in the hidden

123Author's personal copy
728

Neural Comput & Applic (2012) 21:725–734

layer. We term such neurons in the output and hidden layerthe ‘‘target’’ neuron and ‘‘relay’’ neuron, respectively,
corresponding to each input. Note that the relay neurons
are not prescribed before the learning process but are
determined as a result of the learning process, in contrast to
the target neurons.
Thus, the activated synapses are continuously strengthened until a new target is presented, so that the present
target is memorized. This period is called the ‘‘stabilization’’ phase, and the process to learn one I/O mapping,which is composed of a single search phase and stabilization phase, is called the ‘‘learning step.’’ At t*800, we
switch to a new input and the corresponding target pair to

(a)

(b)

Fig. 2 Dynamics of neural activities during learning for two input–
output (I/O) mappings. One learning process for two I/O mappings isshown. I/O mappings are learned in the search phase by the antiHebbian rule (0 \ t \ 350, 800 \ t \ 1200) and in the stabilization
phase by the Hebbian rule (350 \ t \ 800,1200 \ t \ 1700). Here,
we set sNA = 1, sBS = 8, and sFS = 64. As initial states for the
network, we assign the synaptic strength a random value between 0
and 1, except in the case of the ISs, and the neural variable a random
value between 0 and 1. a A raster plot of neurons in the output layer.
In this model, only one neuron in the output layer is active in each oftarget patterns, called the ‘‘target’’ neuron. The target neurons for two
I/O mappings are highlighted by the blue and green bars. The red bar
represents the time in which the neural activity is high (xi [ 0.9).
b Time series of the amplitude of the error signal, E, between the
output activity and target pattern. The blue and green bars above thetime series represent a time in which the first and second (I/O)
mappings are learned, respectively

123

learn this. At that time, the distance between the output
activity and target pattern increases again, and therefore,
the search process progresses on the basis of the antiHebbian rule (800 \ t \ 1,200). In this manner, the neuralactivity can reach the target by switching between theanti-Hebbian and Hebbian rules alternately for synaptic
plasticity, depending on the error. Now, we successively
provide new input–target pairs after the time interval of thestabilization phase, which is sufficiently long for learning
an I/O mapping. See [11] for details about the numericalresults and analysis of the present model.
3.2 Memories as bifurcations
In the present study, the system learns I/O mappings successively, as stated in the above section. Generally, it is noteasy to memorize numerous mappings through successive
learning processes, because after memorizing one mapping, the learning process for the next I/O mapping and theresulting synaptic change may destroy the previous memory. We thus check whether the system can memorize
several I/O mappings in our model, and if possible, we thenanalyze how these mappings are memorized in the neuraldynamical system.
First, we define ‘‘memory’’ based on the viewpoint that‘‘memories as bifurcations.’’ From this viewpoint, the input
is not initial state, but acts as a control parameter thatcauses bifurcation in the neural dynamical system (Fig. 3).
In this paper, we investigate how the dynamical system ischanged as the strength of the input, g, is changed from 0(corresponding to the spontaneous dynamical system) to 1
(corresponding to the evoked dynamical system). If thesystem memorizes an I/O mapping, the evoked dynamicalsystem at g = 1 has a fixed-point attractor that matches thetarget output pattern and is attracted globally from most
initial neural activities. In other words, the attractor ofconcern in the evoked dynamical system has a sufficiently
larger basin. Here, we set the criterion that the basin volume is larger than half of the phase space. Note that thisattractor matching the target pattern need not be an
attractor in the spontaneous dynamical system. In fact,many of the learned targets in the spontaneous dynamicalsystem are not attractors in our model. The nature of the
spontaneous dynamical system is independent of the definition of memory.
By adopting this definition, we analyze the number ofmemories by successively carrying out the learning processes for the I/O mappings. We find that by setting theparameters in our model appropriately, the number ofmemories (memory capacity) is near N, which is the
maximum possible number for the set of I/O mappings in
the sparse coding. In the next subsection, we analyze thisdependence of the number of memories on the parameters,

Author's personal copy
Neural Comput & Applic (2012) 21:725–734

729

Fig. 3 Schematic representation of ‘‘memories as bifurcations.’’ The
lower panels represent the neural dynamics without inputs (spontaneous dynamics) whereas the dynamics upon different inputs (evokeddynamics) are given in the upper panels, where fixed-point attractors
are selected through bifurcations so that the corresponding output
patterns are generated. The learning process changes the neuralsystem both with and without input by the synaptic plasticity. Whenviewing ‘‘memories as bifurcations,’’ there are two types of parameterchanges: one for the input strength (recall process) and the other forthe synaptic plasticity (learning process). The spontaneous andevoked dynamics are modified through the learning process, so thatthe adaptive bifurcation from the spontaneous dynamics to a fixedpoint attractor matching the target is formed upon the input

especially the timescale parameters, sNA, sBS, and sFS(2–4), to find the optimal values.
In order to understand how memories are embedded, weanalyze the dynamical systems after each learning stepwith a fixed pattern of synaptic strength (i.e., the system is
not learning at this time). After learning the given I/O
mappings, the neural activity orbits are examined, first in
the absence of inputs and then in their presence. Figure 4
displays orbits after learning four and seven I/O mappings.
After four I/O mappings are learned (Fig. 4a(i)), the neuralactivity in the output layer in the absence of any input
shows transient itinerant dynamics over three patterns thatare close to three of the target patterns, until it finallyconverges to a fixed point. In general, there can be several
attractors in the neural dynamics, and which attractors arereached depends on the initial conditions of the neuralactivities. However, many orbits from different initialstates also come close to each of the learned targets (see[11]).
In Fig. 4a(ii), an example of neural activity orbits in theabsence of inputs is displayed after seven I/O mappings arelearned. In this case, the attractor is a limit cycle that itinerates over the targets in the cyclic order 2, 6, 3, 4, and 5.
In both cases, the target patterns do not exist as fixed-point
attractors without inputs. Upon the application of the input,
the fixed point or limit cycle collapses, and instead, thecorresponding target pattern becomes a stable fixed-point
attractor. Hence, a memorized output pattern is generated
as a result of bifurcation.
The number of attractors varies through the learning
process. As the learning progresses further, the number of
fixed-point attractors first increases but then decreases as
the learning steps increase beyond three. These attractors

are replaced by one or more limit cycle attractors, while thetotal number of attractors itself decreases. This is because
the limit cycle globally covers the phase space, with little
room left for other attractors (Fig. 4b).
3.3 Dependence of memory capacity on timescales
for synaptic plasticity and neural activity
In this subsection, we discuss the dependence of thememory capacity on the timescale relationships. In ourmodel, there are two timescales for the synaptic plasticity,
in contrast to most neural network models, which only use
one timescale. We focus on the dependence of the memory
capacity on three timescales (including one timescale forneural activity): sNA for changes in the neural activity, andsBS and sFS for the plasticities of the BSs and FSs,
respectively. We study the dependence by varying sBS with
fixed sNA and sFS. Here, we set sNA = 1 and sFS = 64 in
the following analysis.
We measure the memory capacity and find that itreaches the maximum possible number, N, under the sparse
coding case, when the condition sNA  sBS  sFS is satisfied (Fig. 5). This implies that with a single synaptic
timescale (sFS = sBS), as is used with the usual learningmodels, this high capacity is not realized.
This relationship for the timescales requires the following explanation for the optimal learning. In our model,
a smaller value of either sFS or sBS determines the timescale of the search phase, because the search for the target
is possible only on the basis of the change in the flow
structure by the anti-Hebbian rule. Further, sFS determines
the timescale of the memory decay, because the memory
information is embedded in the FSs (5). Because the123Author's personal copy
730

Neural Comput & Applic (2012) 21:725–734

(a)

(b)

Fig. 4 Analysis of changes in flow structure of spontaneous neuralactivity during learning process. a Temporal evolution of neuralactivity in the output layer. The neural activity in the output layer is
projected from the N-dimensional space consisting of the neuralactivities of the neurons in the output layer to a three-dimensional
space by obtaining the products of the output activity and target
output pattern. The gray line represents the orbit in the phase space ofthe dynamical system without inputs, while each colored line
represents the orbit in the phase space in the presence of a different
input. The circles in the phase space represent the targets. (i) Neuraldynamics after learning four I/O mappings. Each axis represents the
product of the neural activity and corresponding target output pattern.
Each of the targets is stable in the presence of the corresponding
input, whereas in the absence of input, it does not exist or is unstable.

(ii) Neural dynamics after learning seven I/O mappings. Each axis
represents the product of the neural activity and corresponding
combined target patterns ui in the figure. The limit cycle is an
attractor in the absence of inputs (the gray line) and is broken in thepresence of each input. Each of the targets is stable in the presence ofthe corresponding input, just as in (i). Some transient orbits with input
starting with the initial states on the broken limit cycle are shown.
b Changes in the number of attractors during the learning process.
The numbers of fixed-point attractors (the green line), limit cycle
attractors (the blue line), and total attractors (the red line) in theabsence of inputs are plotted as a function of the number of learning
steps, i.e., the number of learned targets

timescale for the search phase should be sufficiently
smaller than the memory decay time, sBS  sFS is required
to preserve the previous memory during the search phase.
Next, we consider the relationship between sNA and sBS.
NA
s determines the timescale for the neural dynamics undera given fixed flow structure. If sNA was larger than or of thesame order as sBS, the phase structure would be modified
before the neural activity changes, which would hinder theapproach to the target pattern. In fact, the total search timedecreases rapidly with an increase in the timescale of the
BS plasticity, as will be shown in the next subsection.
Hence, sNA  sBS is required. Accordingly, the relationship sNA sBS  sFS is required for an effective search fora new target without destroying the previous memory.

dynamics during the learning process for the fifth I/Omapping. The time series of neural activities are plotted forthree different timescale values for the backward synapses:sBS = 1 (sNA = sBS  sFS), sBS = 19 (sNAsBS 
sFS, near the peak of the capacity curve), and sBS = 128(sNA  sBS & sFS). Figure 6a displays the raster plot of
the output layer neurons. After successive changes in thefiring of the neurons, only the fifth output neuron keeps on
firing, which corresponds to the target pattern. When theneural activity converges to a target pattern, a single neuron in the hidden layer is activated, which is called a relay
neuron, as defined in the above section. To examine thetransient process before converging to a target, we plot theneural activity of a target output neuron and the corresponding relay neuron in the hidden layer in Fig. 6b. As
shown, there are epochs where both the output and relay
neurons are kept activated. During each of these epochs,
the neural activity pattern stays close to that of the corresponding learned pattern. As shown in the figure, thisresidence at the learned pattern lasts longer as sBS is

3.4 Dependence of learning process on timescales
for synaptic plasticities
We next discuss how the learning process depends on thetimescale relationship. Figure 6 shows the neural activity

123

Author's personal copy
Neural Comput & Applic (2012) 21:725–734

Fig. 5 Memory capacity. Memory capacity is a function of thetimescale for the backward synapses, sBS. The other two timescales,sNA and sFS, are fixed at sNA = 1 and sFS = 64. The values arecomputed from the averages for over 100 learning processes for eachsBS. The error bar indicates the standard deviations. For the learning
processes computed here, we set the synaptic strengths of the FSs andBSs to 0 as initial conditions for the network for simplicity, incontrast to the case in Fig. 2, and set the neural activities at random
values between 0 and 1, just as in Fig. 2. The randomness of theinitial state of the neural activities causes the variability of thelearning process. We also set such initial values in Figs. 7 and 8

increased, while the frequency of visiting each learned
pattern is larger as sBS is decreased.
Now, we will study how the residence time, i.e., theaverage time span for the residence at a target pattern,
depends on the timescales for the synaptic plasticity. This
residence time is related to the memory capacity for thefollowing reason. In the search phase, the output activity
changes from one pattern to another, and the activity may
come close to one of the previously learned target patterns.
Because this pattern differs from the current target pattern,
the attraction to this pattern embedded in the synapses may
be destroyed by the anti-Hebbian rule. In general, thelonger the output pattern stays close to a state corresponding to a previously learned target pattern, the stronger
is this destabilization. Hence, the degree of the destabilization of the previous memory increases with the residence
time at the corresponding pattern.
The dependence of the residence time on timescale sBS
is plotted in Fig. 7. The residence time at each target isdefined as the time span in which the activities of both theoutput target neuron and relay neuron remain larger than agiven threshold, with a value of 0.5 used here (see thecaption of Fig. 6). As shown in Fig. 7, this single residence
time increases roughly in proportion to sBS.
Through the learning process, there are many residencesat the previous targets. Thus, the accumulated residence
time is also dependent on the total search time, which is the731time span required for the search phase through the entire
learning process. The dependence of the total search time
on sBS is plotted in Fig. 7, which shows that it increases
with a decrease in sBS.
Considering the fact that the accumulated residence timeincreases with both the (single) residence time and the total
search time, the accumulated residence time is expected toincrease for both small and large sBS. We compute thistotal accumulated time and also plot it also in Fig. 7. As
shown, it has a sharp drop at the intermediate value ofsBS, at about 16, which agrees well with the optimal valueof sBS required to achieve the maximum memory capacity.
Thus, the timescale relationship for synaptic plasticity forthe optimal memory capacity is explained as the minimum
accumulated residence time.
The residence time at each target depends on the orderin which the target is learned. It is expected that the residence time at an older learned target is short, whereas thatat a newer learned one is long. Thus, we analyze thisdependence. Figure 8 shows the residence times at each of
the learned targets during the learning process of the tenth
I/O mapping. This residence time is plotted for three sBS
values. The residence times for the few targets that werememorized most recently are rather high for sBS = 1 andsBS = 128. During this long residence time, it is easy forthe memories to be destabilized. Thus, only slight traces ofthe earlier targets remain, which is why the residence timeat the earlier learned target is smaller. In fact, the residence
times at the previous target patterns decrease rapidly in thecase of sBS = 1 or 128. In contrast, the residence times in
the case of sBS = 19 remain at almost the same level,
except for the two latest targets. This result shows that agreater record of a learned target remains in the case ofsBS = 19 than for sBS = 1 and 128.

4 Discussion and conclusion
In the present paper, we investigated the dependence of thememory capacity on timescales on the basis of a novel
dynamical system model for the memory [11]. In thismodel, the learning process shapes the ‘‘appropriate’’ flow
structure of the spontaneous neural dynamics throughsuccessive presentations of inputs and their corresponding
outputs. Memory recall is achieved as a result of thebifurcation of neural dynamics from an attractor of the
spontaneous activity to that evoked by the external input
matched with the target pattern. Specifically, we adopted alayered network model with multiple synaptic timescales
and the adaptive reward-penalty (ARP) algorithm. As aresult, we found that an appropriate relationship has to besatisfied among the timescales of the changes in the neuralactivity and that of the plasticities of the FSs and BSs, in123Author's personal copy
732

Neural Comput & Applic (2012) 21:725–734

(a)

(b)

Fig. 6 Neural dynamics for three timescales for synaptic plasticities.
The neural dynamics through the learning process of the fifth I/Omapping are shown for the three timescales for the plasticity of theBSs: sBS = 1, sBS = 19, and sBS = 128. a A raster plot of all theneurons in the output layer. The red bar represents the time during
which the neural activity is high (xi [ 0.5). The vertical axis
represents neuron index i, which is activated at the i-th target, inthe order of learning. Here, the fifth neuron on this index is the target
neuron for the fifth I/O mapping (the magenta raster plot), and thefirst to fourth neurons in the index are the target neurons of thepreviously learned I/O mappings (the red raster plot). In contrast,
the other neurons in the output layer are not yet learned target neurons(the gray raster plot). After the transient time, the fifth neuron

remains activated, which implies that the learning process of thememory is completed. b Neural activities in the hidden and output
layers are shown. Plotted are the neural activities of the first output
neuron (corresponding to the first target neuron) and the ‘‘relay’’neuron in the hidden layer, which is the neuron that is kept activatedin the stabilization phase of learning the corresponding I/O mapping.
(See Sect. 3.1 for a definition of the relay neuron.) In other words, thisneuron receives input from the current active input neuron andoutputs to the target output neuron. Magnified views of the middle
panels are given in the lower panels. We define the residence time as
the time in which the activities of both the target and relay neuron are
higher than a threshold (we set this at 0.5 here). This threshold isrepresented by the dashed line in the lowest panelorder to achieve the maximum number of memorized
patterns. In this section, we discuss the possible relevanceof our results to biological learning: First, the implementation of the present learning mechanism in the biologicalbrain, and second, the functional meaning of the multiple
synaptic timescales.

number and/or type of receptors for the neuraltransmitters in our brain. Hence, the timescales forthe synaptic plasticities are related to the number and/or type of these receptors. When two areas (saybetween the hippocampus and prefrontal cortex) aremutually connected, the forward and backward synaptic connections may have different characteristics,
and hence, their plasticities may differ.
Recall that in our model, a proper relationship has tobe satisfied among the timescales of the changes in
the FSs and BSs and in the neural activity, to achieve
the maximum number of memorized patterns. On thebasis of the above argument, it is suggested that sucha difference in the plasticities may be implemented by
the possible difference between the number distribution and/or types of receptors in the neurons for the

4.1 Biological implementation of learning mechanism
Here, we discuss the possibility of implementing thearchitectures of our model in the brain.
(1)

Multiple timescales: The timescales of the synaptic
dynamics represent the magnitudes of the synaptic
plasticities such as long-term potentiation and longterm depression, and these plasticities depend on the123Author's personal copy
Neural Comput & Applic (2012) 21:725–734

Fig. 7 Total search time, accumulated residence time, and residence
time. The total search time (the green line), accumulated residence
time (the red line), and residence time (the magenta line) throughthe entire learning process are plotted as functions of sBS. The total
search time is the time required for the search phase for all of thelearned I/O mappings, while the residence time is computed using
the average time span for each epoch remaining at a target. The
accumulated residence time is computed by summing the residence
times at the targets memorized previously. These are computed
throughout the entire learning process, averaged over 50 learning
processes733

the magnitude of the error signal. In our brain, neuralmodulators such as dopamine, serotonin, norepinephrine, and acetylcholine may give rise to this error
signal. In particular, dopamine modulates the synaptic
plasticity at the hetero-synaptic connection [13] and isbroadly projected onto the cerebral cortex. Furthermore, the activity of dopamine neurons is related tothe extent to which the response matches the request[14]. Hence, dopamine can act as a global error signal
carrier. Thus, it is suggested that the switching
between positive and negative plasticities, corresponding to that between the Hebbian and antiHebbian rules in our model, is regulated by theconcentration of dopamine.
4.2 Function of multiple timescales
In general, the timescale for memory decay should belarger than that used when searching for the appropriate
neural patterns for a new memory. In our model, the former
timescale is given by the plasticity of the FSs, while thelatter is given by that of the BSs. Indeed, it is necessary in
our model that the timescale for the plasticity of the FSs bemuch larger than that for the BSs.
We expect that this postulate for such multiple timescales is rather general for successive learning. Recently,
Fusi et al. [15] proposed a meta-plasticity-based model,
which may involve multiple timescales similar to those in
our model. They suggested that a meta-plasticity mechanism would be needed for simultaneous robustness andflexibility.
Generalizing our present results, we propose that theeffective plasticity timescales for synapses are different fordifferent roles.
Acknowledgments This work is partially supported by the peopleof Japan, via a Grant-in-Aid for scientific research (No.21120004)
from MEXT, Japan.

Fig. 8 Residence time at each learned target during learning process
for tenth I/O mapping. The residence time during the process to learn
the tenth I/O mapping is plotted against the target index for different
timescales, which are shown in different colors: sBS = 1 (red),sBS = 19 (green), and sBS = 128 (blue). We measure the residence
times at the previous learned targets, i.e., from the first to ninth,during the learning process for the tenth I/O mapping. The target
index order is the same as that for the learning. The residence time iscomputed by averaging over 50 learning processes, in the same way
as used in Fig. 7(2)

forward and backward connections between the given
areas.
Adaptive reward-penalty: In our model, the synaptic plasticity is switched between Hebbian and antiHebbian rules by the ARP algorithm, depending on

References
1. Willshaw DJ, von derMalsburg C (1976) How patterned neuralconnections can be set up by self-organization. Royal Soc Lond
Proc Ser B 194:431
2. Hopfield JJ (1984) Neurons with graded response have collective
computational properties like those of two-state neurons. Proc
Natl Acad Sci USA 81(10):3088
3. Kohonen T (1982) Self-organized formation of topologicallycorrect feature maps. Biol Cybern 43(1):59
4. Barto AG, Sutton RS, Brouwer PS (1981) Associative search
network: a reinforcement learning associative memory. Biol
Cybern 40(3):201
5. Rumelhart DE, Mcclelland JL (1986) Parallel distributed processing: explorations in the microstructure of cognition, vol 1.
Foundations. MIT Press, Cambridge

123Author's personal copy734
6. Luczak A, Bartho P, Marguet SL, Buzsaki G, Harris KD (2007)
Sequential structure of neocortical spontaneous activity in vivo.
Proc Nat Acad Sci 104(1):347
7. Luczak A, Bartho P, Harris KD (2009) Spontaneous events outline the realm of possible sensory responses in neocortical populations. Neuron 62(3):413
8. Fox MD, Corbetta M, Snyder AZ, Vincent JL, Raichle ME (2006)
Spontaneous neuronal activity distinguishes human dorsal andventral attention systems. Proc Nat Acad Sci 103(26):10046
9. Arieli A, Sterkin A, Grinvald A, Aertsen A (1996) Dynamics ofongoing activity: explanation of the large variability in evokedcortical responses. Science 273(5283):1868
10. Mazor O, Laurent G (2005) Transient dynamics versus fixed
points in odor representations by Locust Antennal Lobe projection neurons. Neuron 48(4):661

123

Neural Comput & Applic (2012) 21:725–734
11. Kurikawa T, Kaneko K (2011) Learning shapes spontaneous
activity itinerating over memorized states. PLoS ONE 6(3):e17432. doi:10.1371/journal.pone.0017432
12. Xie X, Seung HS (2004) Learning in neural networks by reinforcement of irregular spiking. Phys Rev E 69(4):41909
13. Jay TM (2003) Dopamine: a potential substrate for synaptic
plasticity and memory mechanisms. Prog Neurobiol 69(6):375
14. Reynolds JNJ, Hyland BI, Wickens JR (2001) A cellular mechanism of reward-related learning. Nature 413(6851):67
15. Fusi S, Drew PJ, Abbott LF (2005) Cascade models of synaptically stored memories. Neuron 45(4):599