平成23年度 電気関係学会四国支部連合大会

51-5

Improvement of Learning Performanceby Recurrent Neural Network with Neurogenesis
Yuta YOKOYAMA, Tomoya SHIMA, Chihiro IKUTA, Yoko UWATE and Yoshifumi NISHIO(Tokushima University)

1. Introduction

3. Simulation Results

In human brain, neuron had been considered to be lostwith age until a few years ago. However, in a recent study,
it was reported that the neurogenesis [1] occurs in thedentate gyrus in the hippocompus. The neurogenesis improves memory and thinking power of the human brain.
In this study, we introduce the neurogenesis into Recurrent Neural Network (RNN). Also, we apply the RNN
with neurogenesis to pattern learning and investigate itsperformance.

We consider the RNN with 100 neurons. The RNN
with neurogenesis leans 5000 times. An input pattern anda teaching pattern are shown in Fig. 2.

2. RNN with Neurogenesis

The existing neurons are randomly replaced by the regeneration neurons in each 1, 5 or 10 learning times till50th learning time. The number of regeneration neurons
changes by 10 steps in learning. After learning process,if diﬀerence value of corresponding position between theRNN with neurogenesis and the teaching pattern is lessthan 0.2, we consider as two patterns is agreement. We
investigate the performance of the RNN with neurogenesisby counting the number of agreement every regenerationneuron positions. The simulation results are summarizedin Tab. 1. This results are obtained from 10 average bychanging the initial condition of RNN.

First, we explain the RNN. The RNN is a neuralnetwork with self-feedback and asymmetric connections.
Back Propagation Through Time (BPTT) is used as theRNN’s learning method. The BPTT was introduced byWilliams in 1989. The BPTT was based on an extension of
the standard Back Propagation (BP) algorithm for a feedforward neural network. The output function described as
Eq. (1). Moreover, the internal state and sigmoid function
which are used for output function are described as eachEq. (2) and (3).
xi (t + 1) = f (ui (t + 1))(1)
ui (t + 1)=

∑wij xj (t)(2)=1
1 + e−u=ηt−1∑

δi (τ + 1)xi (τ )

(4)

τ =t0

where η is the learning rate, τ is the time constant.
We propose the RNN with neurogenesis. Here, the neurogenesis is that some existing neurons are replaced withregeneration neurons in each learning of the RNN (seeFig. 1). Then, the connections of the existing neurons
are regenerated by the neurogenesis. The position of theregeneration neurons are set randomly. And the regeneration connection weights are recasting. After the neurogenesis, the regenerated network learns pattern using BPTT.
self-feedback

pattern.

Table 1: Simulation Results. (a) The number of regenerationNetwork type(a)conventional

0102030405060708090100(3)

where x is the input or output, u is the internal state, wis the connection weight. The updating type of weight is
described as Eq. (4).
∆wij(b)neurons. (b) The number of agreement.

jf (u)(a)Figure 2: Learning pattern. (a) Input pattern. (b) Teaching

RNNwithNeurogenesis

1
32.1
36.5
32
32
32
32
32
32
32
32
32

(b)
timing
5
10
32.1
32.1
72
57
90
72
59.8
83
36.3
90
35.6
80.9
32
55
32.4
38.4
32
36
32
36
32
34

From Tab. 1, the learning performance of the RNN withneurogenesis improves than the conventional RNN in acertain case. And maximum number of agreement is 90.
This value is about three times as many as the conventional RNN’s. Therefore, it is confirmed that the learning
performance of the RNN with neurogenesis depends onfrequency of the neurogenesis and the number of regeneration neurons.

4. Conclusions: new connection: connection: regeneration neuron: existing neuron

Figure 1: RNN with neurogenesis composition.

Powered by TCPDF (www.tcpdf.org)

In this study, we proposed the RNN with neurogenesis
The learning performance of the RNN with neurogenesis
improves than the conventional RNN in the certain case.
References[1] Suzanna Becker, J.Martin Wojtowicz,”A model of hippocampal neurogenesis in memory and mood disorders”,
T rendsCogn, Sci. 11(2), 70-76. 2007.