2013 International Symposium on Nonlinear Theory and its Applications
NOLTA2013, Santa Fe, USA, September 8-11, 2013

Investigation of Influences of Neurogenesis in Multi-Layer Perceptron
Yuta Yokoyama† , Chihiro Ikuta† , Yoko Uwate† and Yoshifumi Nishio††Department of Electrical and Electronic Engineering, Tokushima University
2-1 Minami-Josanjima, Tokushima-shi, Tokushima, 770-8506, Japan
Email: {yuta, ikuta, uwate, nishio}@ee.tokushima-u.ac.jp
Abstract—Neurogenesis is that new neurons are generated in the human brain. We focus on the characteristic of
the neurogenesis with biologically. In the previous study,
we have proposed artificial network model which was applied the neurogenesis to Multi-Layer Perceptron (MLP).
In this study, we show the eﬀectiveness of the proposed
network with neurogenesis for pattern recognition. And weinvestigate the parameter dependency for detailed researchon the influences of neurogenesis on MLP.
1. Introduction
It is said that there are about 10 billion neurons in the human’s brain. The network is formed by connecting of morethan one neuron. However, neurons had been considered
to be lost with age until several years ago. In recent studies, some researchers reported that new neurons are generated in the dentate gyrus of hippocumpus [1]- [3]. This
process is called “neurogenesis”. By utilizing the neurogenesis, some brain cells increase and the network is substantial. It is known that the neurogenesis improves ability
to solve problems by combining new neurons. We focus
on the characteristic of the neurogenesis on biology. In theprevious study, we have proposed artificial network model
which was applied the neurogenesis to Recurrent Neural
Network (RNN) [4] and Multi-Layer Perceptron (MLP)[5]. Therefore, we have researched the performance of theproposed network.
In this study, we investigate the parameter dependency
for detailed research on the influences of neurogenesis on
MLP. Then, we research the learning performance of the
proposed network and focus on the some parameters of theneurogenesis using the proposed network.
2. MLP with Neurogenesis
2.1. Neurogenesis
Before, there is no neural stem cell which makes a neuron in the brain of an adult. It was impossible to generate
the new neuron. Therefore, the neurogenesis had been considered to generate for period of growth. Thus, neurons had
been considered to be lost with age until several years ago.
However, some researchers reported that new neurons aregenerated in the adult brain. This process is called “neurogenesis”. The neurogenesis in the hippocumpus of thehuman brain was discovered in the late 1990s by Erickson
et al [1]-[3]. The neurogenesis is included in the existingneural circuit by given the learning and new neurons aregenerated in the human brain. It is known that the neurogenesis improves memory, learning, thinking ability, and soon. We focus attention on the characteristic of the neurogenesis.
2.2. Rule of Neuron Updating
We use a Multi-Layer Perceptron (MLP) which is comparatively easy network for artificial neural network andone of a feed-forward neural network. The MLP is a most
famous feed-forward neural network. This network is used
for pattern recognition, time series prediction, noise reduction, motion control, and other tasks. The MLP is composed some layers of neuron, it has input layer, hidden
layer, and output layer. This network learns to the tasks
by changing the weight parameters. Generally, the performance of the MLP is changed by the number of neurons.
Moreover, we used the Back Propagation (BP) which is oneof the MLP’s learning method.
A Back Propagation (BP) is used to the MLP’s learningalgorithm. The BP was introduced by D.E. Rumelhart in1986 [6]-[8]. In this algorithm, the network calculate theerror from the output and teaching signal.
The neuron has the multi inputs and a single output. The
updating rule of neuron is described by Eq. (1).
 n∑(1)xi (t + 1) = f  wi j (t)x j (t) − θ ,jwhere x is the input or output and w is the connection
weight parameter and θ is threshold. In this equation, theweight of connection and threshold of neuron are learned
by BP algorithm. We used the sigmoid function for the output function. The error of MLP propagates backward in thefeed-forward neural network. BP algorithm changes valueof weights to obtain smaller error than before. The totalerror E of the network is described by Eq. (2).
1 ∑∑(t pi − o pi )2 ,2 p=1 i=1pE=n(2)where E is the error value, p is the number of the input data,n is the number of the neurons in the output layer, t pi is the- 382 -value of the desired target data for the pth input data, ando pi is the value of the output data for the pth input data. The
function of the connection weight is described by Eq. (3).

∆ p wk−1,k= ηkp j ok−1
pi = −ηij∂E p∂wk−1,ki, j,(3)where wk−1,kis the weight between the ith neuron of thei, jlayer k − 1 and the j the neuron of the layer k, and η is theproportionality factor known as the learning rate.

of periodic behavior with period 3 and burst part of spread
points over the invariant interval). Moreover, in cases
where the parameter α is 4.0, it is the fully chaotic time series. In this study, we consider y = 0.6 as basis. In this timing method, the neurogenesis is occurred when the value ofy takes the range between 0.6 to 0.7.
3. Simulations
We consider a pattern recognition, where 26 alphabets
patterns consisting of 0 and 1 in Fig. 2 are stored in theneural network for recognition.

2.3. Network structure
In this study, we apply the behavior of neurogenesis tothe MLP. We have proposed artificial network model which
was applied the neurogenesis to MLP [5]. Figure 1 showsa structure of the proposed network.

Figure 2: Input patterns.
We consider that the propose network is composed ofthree layers. The number of neurons in the input layer is35, and the output layer is 26. In the proposed MLPs, weset 10 neurons in the hidden layer at the start of learning.
Therefore, we set the maximum number of the neurons inthe hidden layer. The MLPs learn 5000 times during theone trial. The learning rate is η = 0.005 and initial values ofthe weights are between −0.5 and 0.5 at random. Moreover,
we compare the learning performance of the following fivekinds of MLPs:
1. The conventional MLP

Figure 1: MLP with neurogenesis.

2. The MLP with random neurogenesis

In fact, the timing of generation new neurons are lesscertain. Then, we consider two kinds of the neurogenesis.
One is the periodic neurogenesis. In the case of the periodic neurogenesis, new neurons generated at every 10 iterations during the learning process. The other is the chaoticneurogenesis. In the case of the chaotic neurogenesis, newneurons generated by using the logistic map. Hasegawa etal. investigated solving abilities of the Hop-field NN withvarious surrogate noise, and they concluded that the eﬀects
of the chaotic sequence for solving optimization problems
can be replaced by stochastic noise with similar autocorrelation [9]. The updating function of the logistic map isdescribed by Eq. (4).
y(n + 1) =αy(n)(1 − y(n)).

(4)

If the parameter α is changed, Eq. (4) will be served inchaos through period-doubling bifurcation. In this study,
we use that the parameter α = 3.82740 and 4.0. When
we choose that the parameter α is 3.82740, it is wellknown that the map produces intermittent bursts just before periodic-windows. In cases where the parameter α is3.82740, it is the intermittently chaotic time series. Thistime series could be divided into two phases (laminar part

3. The MLP with periodic neurogenesis
4. The MLP with intermittently chaotic neurogenesis(α=3.82740)
5. The MLP with fully chaotic neurogenesis (α=4.0)
In the MLP with random neurogenesis, new neurons are
introduced at random in the hidden layer. We use a Mean
Square Error (MSE) as the measure of their performances.
MSE is defined by Eq. (5).
1 ∑(tn − on )2 .
N n=1
NMS E

=(5)

We make a comparison between the performance of theconventional MLP and the proposed MLPs. Moreover, we
carry out 100 trials with diﬀerent initial weights of connections.
3.1. Learning Performance
In this section, we compare the learning performance ofthe five diﬀerent MLPs. We show an example of the learning performance of the MLPs in Fig. 3. In this example, we- 383 -set that the conventional MLP has 30 neurons in the hidden
layer. Therefore, the proposed MLPs are set that the number of neurons in the hidden layer increases until 20 during
the learning.

Table 1: Learning performance.
(a) The number of neurons in hidden layer. (b) Average of processing
time. (c) Average of error. (d) Minimum error. (e) Maximum error. (f)St. Dev.

Figure 3: Learning performance of five MLPs.
From Fig. 3, the error of each MLP decreases. Moreover,
we focus on the number of the neuron in the hidden layer.
Therefore, we set that the conventional MLP is 30 neurons
in the hidden layer. The proposed MLPs are set that thenumber of neurons in the hidden layer is 20 neurons at thestop of learning. However, we can see that the learning
performance of the proposed MLPs are as well or better
than the conventional MLP.
Therefore, we used the conventional MLP of 10, 20, 30,
40 and 50 neurons in the hidden layer. During the learning, the number of neurons in the hidden layer is increased
until 20 and 30. We show the average of the learning performance of the MLPs in Table 1.
From Table 1 (1), we show the learning performanceof the conventional MLP. We considered the best learning
performance when we set 30 neurons in the hidden layer.
Namely, we can say that 30 neurons is the best number ofthe neurons in the hidden layer. Therefore, we compare theconventional MLP and the proposed MLPs with 30 neurons in the hidden layer. From Table 1 (2), (3), (4) and (5),we show the learning performance of the proposed MLPs.
We were able to obtain the performance of the proposed
MLPs with 20 neurons as well or better than the conventional MLP with 30 neurons in the hidden layer. At thesame time, we were able to obtain that the average of theprocessing time of the proposed MLPs are faster than theconventional MLP because used the small number of neurons in the hidden layer. Thus, we considered that the network of the quick processing time and high ability has been
proposed. Because, we considered that the parameter of theconnecting weight changed by the neurogenesis.

(a)102030
4050

(b)
1.23
2.14
3.04
3.98
4.83

(1) The conventional MLP.
(c)
(d)
(e)
0.008130.003410.01460
0.003210.00084
0.00747
0.002820.00051
0.00809
0.002960.00045
0.00784
0.003300.00035
0.00910(f)0.002180.001540.001610.00174
0.00201(a)
2030(2) The MLP with random neurogenesis.
(b)
(c)
(d)
(e)
2.77
0.003070.00085
0.00670
3.65
0.002440.00055
0.00673(f)
0.001400.00142(a)
2030(3) The MLP with periodic neurogenesis.
(b)
(c)
(d)
(e)
2.07
0.00306
0.000840.00948
2.83
0.002480.00055
0.00662(f)
0.00160
0.00137

(a)
2030
(a)
2030

(4) The MLP with intermittently chaotic neurogenesis.
(b)
(c)
(d)
(e)
(f)
2.13
0.00288
0.00095
0.008180.001482.96
0.002440.00060
0.007920.00156(5) The MLP with fully chaotic neurogenesis.
(b)
(c)
(d)
(e)
2.13
0.002920.000840.00670
2.97
0.002340.000590.00792(f)
0.001210.00152

From Table 2, we focus on the conventional MLP with30 neurons in the hidden layer. During the learning, thenumber of neurons in the hidden layer is increased until 20
at the stop of learning. We compare the learning performance of the proposed MLPs with the conventional MLP.
We were able to obtain the performance of the proposed
MLPs with 20 neurons as well or better than the conventional MLP with 30 neurons in the hidden layer. Thus,
we were able to obtain the good performance which is thesmall number of neurons in the hidden layer.
Table 2: Average of pattern recognition performance.
(a) The number of neurons in hidden layer. (1) The conventional MLP.
(2) The MLP with random neurogenesis. (3) The MLP with periodicneurogenesis. (4) The MLP with intermittency chaotic neurogenesis.
(5) The MLP with fully chaotic neurogenesis.
(a)
2030

(1)
95.15
95.68

(2)
94.70
98.05

Accuracy rate
(3)
(4)
95.44
97.66
98.20
97.66

(5)
97.07
98.49

3.2. Recognition Performance

3.3. Parameter Dependency

In this section, we evaluate the recognition performance.
Table 2 shows the average of pattern recognition of the 26
input patterns.

In this section, we investigate the parameter dependency.
We research change of some parameters by the proposednetwork.

- 384 -

3.3.1. Change of the parameter of the logistic map
We show the learning performance as change of α. α isthe parameter of the logistic map. We show results of thechange of α in Figs. 4 and 5.

3.3.3. Change of the number of the neurogenesis by onegeneration
We show the learning performance as each number ofthe neurogenesis by one generation. When this parameteris large, more neurons are introduced in one generation.
Table 3: Learning performance as diﬀerent the number of the neurogenesis by one generation.
(a) The number of neurons by one generation. (b) Average of error.
(a)
1
2
5
10(b)
0.00288
0.00287
0.00257
0.00248

From Table 3, we can say that the error decreases byincreasing the number of the generated neurons.
Figure 4: Learning performance as diﬀerent α (between 3.7 and 3.825).
4. Conclusions

Figure 5: Learning performance as diﬀerent α (between 3.875 and 4.0).
Figures 4 and 5 strike oﬀ the solution of the period 3 forthe logistic map. From these figures, we can see that the error decreases in the vicinity of the solution of the period 3.
However, we can not see the substantial change.
3.3.2. Change of the starting learning time of the neurogenesis
We show the learning performance as changing the startof learning time of the neurogenesis. When this parameteris small, the generation of neuron is quickly introduced.
From Fig. 6, we can see that the learning time of thequick start is better than the learning time of the slowly.

Figure 6: Learning performance as diﬀerent the start learning iterationof the neurogenesis.

In this study, we showed the eﬀectiveness of the proposed network with neurogenesis for pattern recognition.
And we investigated the parameter dependency for detailed
research on the influences of neurogenesis on MLP. We
compared the five kinds of MLPs by pattern recognition.
We investigated in some approaches. Then, we were able
to obtain the good performance by generating new neurons in the hidden layer during the learning process. However, we were able to obtain the learning performance alsochanged by the timing and number of neurogenesis.
Acknowledgment
This work was partly supported by MEXT/JSPS Grantin-Aid for JSPS Fellows (24·10018).
References[1] S. Becker, J. M. Wojtowicz, “A Model of Hippocampal Neurogenesis in Memory and Mood Disorders,” Cognitive Sciences, vol. 11, no. 2, pp. 70-76, 2007.
[2] R. A. Chambers, M. N. Potenza, R. E. Hoﬀman, W. Miranker, “Simulated
Apotosis/Neurogenesis Regulates Learning and Memory Capabilities of Adaptive Neural Networks,” Neuropsychopharmacology, pp. 747-758, 2004.
[3] H. Satoi, H. Tomimoto, R. Ohtani, T. Kondo, M. Watanabe, N. Oka,
I. Akiguchi, S. Furuta, Y. Hirabayashi and T. Okazaki, “Astroglial Expression of Ceramide in Alzheimer’s Disease Brains: A Role During Neuronal
Apoptosis,” Neuroscience, vol. 130, pp. 657-666, 2005.
[4] Y. Yokoyama, T. Shima, C. Ikuta, Y. Uwate and Y. Nishio, “Improvement ofLearning Performance of Neural Network Using Neurogenesis,” Proceedingsof RISP International Workshop on Nonlinear Circuits and Signal Processing(NCSP’12), pp. 365-368, Mar. 2012.
[5] Y. Yokoyama, C. Ikuta, Y. Uwate and Y. Nishio, “Performance of Multi-Layer
Perceptron with Neurogenesis,” Proceedings of International Symposium on
Nonlinear Theory and its Applications (NOLTA’12), pp.715 718, Oct. 2012.
[6] D.E. Rumelhart, G.E. Hinton and R.J. Williams, “Learning Representations byBack Propagation Error,” Nature, vol. 323-9, pp. 533-536, 1986.
[7] D.E. Rumelhart, G.E. Hinton and R.J. Williams, “Learning Internal Representations by Error Propagation,” Parallel Distributed Processing, vol. 1, pp. 318362, 1986.
[8] D.E. Rumelhart, J.L. McClelland, and the PDP Research Group, “Parallel distributed processing,” MIT Press, 1986.
[9] M Hasegawa, T Ikeguchi, T Matozaki and K Aihara, “An analysis on additive eﬀects of nonlinear dynamics for combinatorial optimization,” IEICE
Trans. Fundamentals, vol. E80-A, no. 1, pp. 206-213, 1997.

- 385 -