2012 International Workshop on Nonlinear Circuits,Communications and Signal Processing
NCSP'12, Honolulu, Hawaii, USA. March 4-6, 2012

Improvement of Learning Performance of Neural Network
Using Neurogenesis
Yuta Yokoyama, Tomoya Shima, Chihiro Ikuta, Yoko Uwate and Yoshifumi Nishio
Tokushima University
2-1 Minami-Josanjima, Tokushima, Japan
Phone:+81-88-656-7470
FAX:+81-88-656-7471
Email: {yuta, s-tomoya, ikuta, uwate, nishio}@ee.tokushima-u.ac.jp

Abstract

2. Back propagation Through Time (BPTT)

Neurogenesis is that new neurons are generated in the humanbrain. The new neurons create new network. The neurogenesis causes the improvement of memory, learning, thinking
ability, and so on. We consider that the neurogenesis can beapplied to an artificial neural network.
In this study, we propose the Recurrent Neural Network(RNN) with neurogenesis and apply to pattern learning. In
the RNN with neurogenesis, some neurons are replaced withregenerated neurons. We compare the learning performanceof the RNN with neurogenesis with the conventional RNN.

A Back Propagation Through Time (BPTT) is used to theRNN’s learning method [6][7]. The BPTT was introduced byR.J. Williams and D. Zipser in 1989 [8]. The BPTT algorithm
is based on the standard BP algorithm of feed-forward neuralnetwork. Figure 1 shows the point of view of the BPTT. Figure 1 (a) shows the example of the RNN which is composedof 3 neurons with 2 input. It is possible to replace the networkof Fig. 1 (a) on the network of Fig. 1 (b). Figure 2 shows thediagram of error propagation of real time BPTT.
The updating rule of neuron is described by Eq. (1).
∑xi (t + 1) = f wij xj (t) ,(1)1. Introductionj

In the human brain, neurons had been considered to be lostwith age until several years ago. However, in recent studies, some researchers reported that new neurons are generated
in the dentate gyrus of hippocumpus [1][2]. This process iscalled “neurogenesis”. It is reported that this process occur
all human brains. It is known that the neurogenesis improves
memory and thinking ability by connection of new neurons.
In this study, we apply the behavior of neurogenesis to theRecurrent Neural Network (RNN) which is an artificial neural network. The composed neurons of the RNN have selffeedback and are connected asymmetrically [3]-[5]. Namely,
we can say that the property of the RNN structure is similar
to the biological neural network.
In the proposed neural network, some existing neurons arereplaced to new neurons by effect of neurogenesis. We namethis network “RNN with neurogenesis.” In order to confirm the efficiency of neurogenesis, we investigate the performance of RNN with neurogenesis for learning several alphabet patterns. By using learning method of Back Propagation
Through Time (BPTT), we confirm that the RNN with neurogenesis obtains better results than the conventional RNN.

where f is the sigmoid function, x is the input or the output and w is the connection weight. The sigmoid function isdescribed by Eq. (2). This is used for the output function.
f (a)

=1.
1 + e−a(2)Moreover, the complementary error function and the conditional equation are described by Eqs. (3), (4) and (5).
J(tl , t) =−t
1 ∑ ∑ei (τ )2 ,2 τ =t +1 i(3)l{ei (t)=di (t) − xi (t)0tl= t − α.

if i ∈ O
otherwise,(4)(5)

where d is the teaching signal, O is the set of output unit, tl is
the time to initialize network state using new patterns and α- 365 -∆wij

δi (τ + 1)xi (τ ),

…

a

B

…

b

C

…

c

D

…

d(6)

τ =tl


 δi (t) = f 0 (si (t))ei (t)∑δi (τ ) = f 0 (si (τ ))(ei (τ ) + j wji δj (τ − 1)
if τ < (t − 1),A

The number ofITERATIONSPatterns

t−1∑= ηPatterns

is the time between the input layer and the output layer. The
updating type of connection weight is described by Eqs. (6)and (7).

A…

aB

…

b

C…

c

D…

dThe number ofITERATIONS(7)EPOCH 1

EPOCH 2where η is the learning rate, δ is the back propagation term,
Figure 3: Explanation of ITERATION and EPOCH.
si is the internal state and τ is the time constant.
Figure 3 shows explanation of ITERATION and EPOCH.
We define ITERATION and EPOCH to use the BPTT algoThe neurogenesis in the hippocumpus of the human brainrithm. ITERATION is the number of learning of one patternwasdiscovered in the late 1990s by Erickson et al. Beforein EPOCH. EPOCH is the number of patterns and ITERAthattime,
the neurons had been considered to be lost withTION.
age. The neurogenesis is that new neurons are regenerated in
3. RNN with Neurogenesis
the human brain. The neurogenesis causes to improve memory, learning, thinking ability, and so on. We believe that toadapt the neurogenesis to the RNN can be effective for improve learning performance, and propose the RNN with neurogenesis.
We explain the RNN with neurogenesis. Figure 4 shows astructure of the RNN with neurogenesis. In this system, existing neurons are randomly chosen and removed, and then
new neurons are regenerated to the removed neurons. At theInput signal
same time, all the weights connecting to the regenerated neu(a)rons are newly set between -0.5 and 0.5 at random. In thisstudy, the process to regenerate neurons and connection is“neurogenesis.” After that, the connection weights are newly
Unit ofthe RNN
calculated.

Input pattern
Initialtl

t-1tTime(b)Figure 1: The composition of the RNN for the BPTT logic.
TeachingpatternUnit ofthe RNN: Connection: New connection: Existing neuron: Regenerated neuron

Input pattern
tltl+1

t-2

t-1

tTime

Figure 4: The model of the RNN with neurogenesisn.

Figure 2: The BPTT algorithm.

- 366 -

100

The number of agreement [%]90

The number of agreement [%]4. Simulation Results

90

80
In this study, we consider the RNN with 100 neurons. The
70
time between the input layer and the output layer is α = 10.
(a)-(e)60
We make a comparison between the performance of the con(b)-(f)50ventional RNN and the proposed the RNN with neurogen(c)-(g)40(d)-(h)30esis. Figure 5 shows input patterns and teaching patterns.
20In this study, we prepare 4 learning patterns. We randomly
10compose 100 each network and calculate average of the num0
ber of agreement between output patterns and teaching pat0
100200300400terns. Here, if different value of corresponding position beLeaning time
tween output patterns and teaching patterns is less than 0.2,
we consider as agreement. We examine the performance of Figure 6: The simulation results of the conventional RNN (average :the conventional RNN and the RNN with neurogenesis using 75.0% ).
two simulations.
100(a)

(b)

(c)

(d)

807060(a)-(e)50(b)-(f)40(c)-(g)30(d)-(h)

20
1000

100

200

300

400

Leaning time

(e)

(f)

(g)

(h)

Figure 5: Learning patterns. (a)-(d) Input patterns. (e)-(h) Teaching pat-

Figure 7: The simulation results of the RNN with neurogenesis (average :79.5% ).

terns.

4.1. Influence by the Neurogenesis

neurons increases by 10 or the regenerated neurons are set atEPOCH 2, 4, 6, and 8. And Fig. 8 is visualization of Tab. 1.

In this simulation, we show that the neurogenesis has aninfluence on the network. The number of learning times is Table 1: Performance of the RNN. (A) The number of regeneration neu400, EPOCH is 20 and ITERATION is 20. We introduce the rons. (B) The number of agreement.
10 regenerated neurons by 2 steps till EPOCH 18. The RNN
Network type(A)(B) [ % ]and the RNN with neurogenesis learn each pattern.
TimingFigure 6 shows the performance of the conventional RNN.
246
8And, Fig. 7 shows the performance of the RNN with neurogeConventional
070.60
70.60
70.60
70.60
nesis. From these results, the performance of the RNN with
1072.16
71.33
71.55
71.31
20
71.96
71.19
70.89
70.30
neurogenesis is difference from the RNN. Therefore, we can
30
71.66
70.28
70.72
69.61
say that the neurogenesis is had influence on the RNN.
4.2. The Performance of the RNN with Neurogenesis

RNN
withneurogenesis

40
50
60
70
80
90
100

71.49
71.99
72.48
71.59
70.81
71.22
71.56

71.09
70.47
69.69
70.72
70.01
69.59
70.11

70.88
70.36
70.94
69.8270.76
70.61
69.75

70.10
70.26
69.14
68.68
68.35
68.48
67.91

In this simulation, we examine the performance of theRNN with neurogenesis depending on the number of the regenerated neurons and the timing of set of regenerated neurons. The number of learning times is 200, EPOCH is 10 and
From these results, we consider that the performance ofITERATION is 20. The RNN and the RNN with neurogenesis learn 4 learning patterns. Table 1 shows the performance the RNN with neurogenesis is better than the conventionalof the RNN with neurogenesis if the number of regenerated RNN in a certain case. Especially, we can say two things.

- 367 -The number of agreement [%]Acknowledgment
This work was partly supported by The Nakajima Foundation.
References[1] S. Becker and J. Martin Wojtowicz, “A model of hippocampal neurogenesis in memory and mood disorders,”
Cognitive Sciences, vol. 11, no. 2, pp. 70-76, 2007.
[2] R. A. Chambers, M. N. Potenza, R. E. Hoffman andW. Miranker, “Simulated apotosis/neurogenesis regulateslearning and memory capabilities of adaptive neural networks,” Neuropsychopharmacology, pp. 747-758, 2004.

The number of neurogenesis

Figure 8: The simulation results.

[3] S. Ismail and M. Ahmad, “Recurrent neural network
First, even if the timing of set of the regenerated neurons iswith back propagation through time algorithm for Arachanged, the performance of the RNN with neurogenesis isbic recognition,” Proceedings 18th European Simulation
better than the conventional RNN for the number of the reMulticonference, pp. 98-102, 2004.
generated neurons 10. In this instance, the best performanceis 72.16% . This value improved about 2.0% as many as the [4] R. J. Williams and J. Peng, “An effect gradient-baced algorithum for on-line training of recurrent network trajecconventional RNN. Second, even if the number of the retories,” Neural Computation, pp. 490-501, 1990.
generated neurons is changed, the performance of the RNN
with neurogenesis is better than the conventional RNN for the[5] P. J. Werbos, “Back propagation through time: What ittiming of set of the regenerated neurons 2. In this instance,
does and how to do it,” Proceeding of the IEE, no. 10,
the best performance is 72.48% . This value improved about
pp. 1550-1560, Oct. 1990.
2.5% as many as the conventional RNN.
Therefore, if the timing of set of the regenerated neurons or [6] K. Doya and S. Yoshizawa, “Adaptive neural oscillator
using continuous-time back-propagation learning,” Neuthe number of regenerated neurons exceed a certain value, it
ral Networks, vol. 2, pp. 375-385, 1989.
is very highly possible that the performance of the RNN withneurogenesis is less than the conventional RNN. Namely, in[7] D. Rumelhart, D. Hinton and G. Williams, “Learning inthis study, we can assume that the optimal number of theternal representations by error propagation,” Parallel Disregenerated neurons to achieve the best performance of thetributed Processing, M.I.T. Press, MA, vol. 1, pp. 318RNN with neurogenesis exists.
362, 1986.
[8] R. J. Williams and D. Zipser, “Gradient-based learningalgorithms for recurrent connectionist networks,” Tcech.
Report NU-CCS-90-9, Northeastern University, 1990.

5. Conclusion
In this study, we applied the behavior of neurogenesis to
the Recurrent Neural Network (RNN) which is an artificial
neural network. The composed neurons of the RNN haveself-feedback and are connected asymmetrically. Namely, wecan say that the property of the RNN structure is similar tothe biological neural network.
In the proposed neural network, some existing neurons are
replaced to new neurons by effect of neurogenesis. We named
this network “RNN with neurogenesis.” In order to confirm
the efficiency of neurogenesis, we investigated the performance of RNN with neurogenesis for learning several alphabet patterns. By using learning method of Back Propagation
Through Time (BPTT), we confirmed that the RNN with neurogenesis obtains better results than the conventional RNN.

- 368 -