IEEE Workshop on Nonlinear Circuit Networks
December 12-13, 2014

Investigation of Multi-Layer Perceptron with Pulse
Glial Chain Including Neurogenesis
Chihiro Ikuta

Yoko Uwate

Yoshifumi Nishio

Dept. of Electrical and Electronics Eng., Dept. of Electrical and Electronics Eng., Dept. of Electrical and Electronics Eng.,
Tokushima University
Tokushima University
Tokushima University
2–1 Minami–Josanjima,
2–1 Minami–Josanjima,
2–1 Minami–Josanjima,
Tokushima Japan
Tokushima Japan
Tokushima Japan
Email: ikuta@ee.tokushima-u.ac.jp
Email: uwate@ee.tokushima-u.ac.jp
Email: nishio@ee.tokushima-u.ac.jp

Abstract—In this study, we propose Multi-Layer Perceptron(MLP) with pulse glial chain including neurogenesis. In thisnetwork, we one-by-one connect glia unit with neurons ina hidden-layer. The glia generates a pulse according to theconnecting neuron output. This pulse increases the thresholdof the neuron and excites the neighboring glias. The pulse
generation frequency is also changed according to the connecting
neuron output. Moreover, we introduce a neurogenesis to the
network. By the neurogenesis, any neurons are removed andnewborn neurons are set to same position. The removed neurons
are chosen by the number of pulse generation of the glia, andthe newborn neuron has random value in the connection withother neurons. We consider that the unimportant neurons forthe learning are removed and the newborn neurons become theimportant neurons. By computer simulations, we confirm that the
proposed MLP obtains a better solving ability than the previous
MLP.

of inactivity is dynamically changed by the continuous stimulus from the connecting neuron. In this model, we introduce
a neurogenesis to the MLP. The glia count the number ofpulse generations. When the number of pulse generations are
small number, the connecting neuron is removed. After that,the newborn neuron set to the same position as the removed
neuron. By the frequency of the pulse generation, the glia canfind the important neurons and the unimportant neuronsn. By
the neurogenesis, we consider that the number of contributory
neurons for the network performance increase. We confirm
that the performance of the proposed MLP improves than theprevious MLP, moreover we show the characteristics of the
proposed MLP.

I. I NTRODUCTION

The MLP is a famous feed forward neural networks. In
general, the weight of connection is tuned by a Back Propagation (BP) algorithm [6]. The BP algorithm is useful forthe learning of the MLP, however this algorithm often falls
into a local minimum. In the proposed MLP, we connect theglias with the neurons. The glia is excited by the output of
the connecting neuron. Then, the glia generates the pulse.
This pulse is input to the connecting neuron threshold andis transmitted to the neighboring glias. The neighboring glias
are excited by the transmitted pulse and also generate thepulse. Thereby, the pulse is transmitted into the glial network.
Moreover, the period of inactivity of the glia becomes shorter
when the glia is excited in continuity. By this process, each glia
has a different pulse generation cycle. In the proposed MLP,
we introduce the neurogenesis into the neurons in the hiddenlayer. The neurogenesis happens at a regular iteration. The
removed neuron is chosen by the number of pulse generationsof the connected glia. If the number of the pulse generationof the connected glia is smaller than the decided value, theconnecting neurons are removed and newborn neurons areconnected in the same position. The weight of connection of
the newborn neuron is decided at random.

A glia and a neuron are existing in a human brain. The
neuron has been considered to an important cell for a higher
brain function. Actually, the neuron can transmit electric signal
each other and conduct advanced information processing. On
the other hand, the glia was considered to the supporting cellfor the neuron. The glia exists between the neurons and a bloodvessel and provides necessary nutrients which are carried into
the blood vessel to the neurons. Recently, some researchersreported that the glia controls an ion concentration [1]-[3].
The glia uses various kinds of ions such as a glutamate acid,an adenosine triphosphate, a Ca2+ , et al. [4]. Among them,
we focus on the Ca2+ . The glia occurs a Ca2+ concentration
wave and this wave transmits to other glias, moreover the Ca2+
increases a membrane potential of the neuron. Hence, the glia
closely relates to the higher brain function.
In this study, we propose a MLP with pulse glial chain
including neurogenesis 1 which is extended from the MLP
with pulse glial chain based on individual inactivity period[5]. The glias are one-by-one connected with the neurons. The
glia receives the connecting neuron output and is excited bythis output. The excited glia generates a pulse and this pulse
transmits to other glias and the neurons. Moreover, the period
1 We have already reported this method in Proc. of NOLTA’14, pp. 508-511,
Sep. 2014. In this study, we add new results to this paper.

II. P ROPOSED MLP

A. Updating rule of neuron
We show a proposed updating rule of the neuron. We add theglial pulse to the threshold of neuron. We use this updating rule

- 70 -

j=1

where y is an output of the neuron, w is a weight ofconnection, x is an input of the neuron, and θ is a thresholdof neuron α is a weight of the glial effect. We can change
the glial effect by change of α. In this equation, the weightof connection and the threshold are changed by BP algorithm.
The glial is independent from BP algorithm, thus the weightof the glial effect is not changed by BP algorithm. On theother hand, we use the standard updating rule of the neuron
in the input-layer and the output-layer.
B. Glial pulse
The glia has a response to the output of the connecting
neuron. The glia response is described by Eq. 2.

1

50

0.8

40

0.6

30

0.4

20

0.2

10

θg

ψ(t)

to the neurons in the hidden layer. It is described by Eq. (1).
n∑yi (t + 1) = f wij (t)xj (t) − θi (t) + αψi (t) ,(1)

Glial pulse
θg

0

0
0

100

200
Iteration

300

400

Fig. 1. The correlation between the pulse generation and the period ofinactivity.

the neurons in the hidden-layer. We count the number of
excitations of the glia. If the number of excitations of theglia is smaller than the constant value, the neuron is removed
and the newborn neuron is connected in the same position.
We give the random value to every weight of connection of
the newborn neuron.

ψi (t + 1) ={1, {(θn < yi ∪ ψi+1,i−1 (t − i ∗ D) = 1)
∩ (τi ≥ θgi )} , (2)γψi (t), else,
where ψ is an output of a glia, i is a position of the glia, θn
is a glia threshold of excitation, y is an output of a connected
neuron, D is a delay time of a glial effect, τ is local time of theglia during a period of inactivity, θg is a length of the period
of inactivity, γ is an attenuated parameter. When the output of
the connecting neuron is larger than the excitation thresholdof the glia, the glia generates pulse. Then, the output of theglia ψ has 1. After that the pulse decreases in an exponential
fashion. The pulse excites the neighboring glia which has adelay of the iteration D.
In this model, the glia has a dynamic period of inactivity.
The dynamic period of inactivity is described by Eq. (3).
θgi (t + 1) ={θgi (t) − 1, ψgi (t) = 1 ∩ ψgi (t − θgi ) = 1.
, (3)θgi (0), else,
If the glia continuous generates the pulse, the length of theperiod of inactivity becomes shorter. In other case, if the glia
stops the pulse generation, the length of the period of inactivity
becomes a original value. By the change of the period ofinactivity, each glia has different period of inactivity, thereby
the glial network obtains the various pulse generation pattern.
We show an example of the change of the period of inactivity
in Fig. 1. In this figure, we use θg (t) − 5 in each change stepof the period of inactivity for an understandability. The pulse
generation frequency becomes shorter according to the changeof the period of inactivity.
C. Neurogenesis
The neurogenesis happens into the adult human brain, moreover some researchers reported that the connecting position of
the newborn neuron is decided by the glia [7]. In the proposedmodel (shown as Fig. 2), we introduce the neurogenesis toFig. 2.

Neurogenesis.

III. S IMULATIONIn this study, we use a Two-spiral Problem (TSP) for thesimulation task.The TSP is a famous task for the artificial
neural network and has a high nonlinearity [8] [9]. In thissimulation, we use 130 spiral points for the learning. We
input the coordinates of the spirals, and the MLP learns theclassification of the spirals. In the TSP, the difficulty and thenonlinearity increase according to the number of spiral points.
In this simulation, we compare the three kinds of the MLPs
which are;(1) The standard MLP.
(2) The MLP with pulse glial chain based on individual
period of inactivity.
(3) The MLP with pulse glial chain including neurogenesis.
The standard MLP does not have an external unit, thus this
MLP often falls into a local minimum. The MLP with pulse
glial chain based on the individual period of inactivity is
the previous model which was proposed in WCCI’14. We
use a Mean Square Error (MSE) for the evaluation of theperformance.
A. Learning performance
We obtain the result from 100 trials, and one trial has
100000 iterations. In every trial, we give different initialcondition. In this simulation, the neurogenesis happens at
50000 iterations. Then, if the number of pulse generations issmall, the neuron is removed and we set a newborn neuron.
From the simulation result, we obtain four kinds of evaluation- 71 -

MLP economizes the neurons than the other MLPs by theneurogenesis in the large number of neurons.
1

0.1

Average
0.09604
0.00904
0.00827

Minimum
0.000270.000370.00087

Maximum
0.23087
0.046850.04633

0Std. Dev.
0.05751
0.011610.00975

Fig. 4.

0.01

MSE

0.008
0.006
0.004
0.002
0
30

Number of removed neurons

Fig. 3.

40

60

80

Dependency of number of neurons.

IV. C ONCLUSIONS

0.012

20

20

Number of neurons

Next, we obtain the average errors in different number ofremoved neurons which is shown in Fig. 3. The average error
decreases to 15 removed neurons after that the average error
increases. From this result, we can say that many unimportantneurons exist in the network and the unimportant neurons are
effectively removed by the neurogenesis.

10

(3)

0.001

B. Dependency on the number of removed neurons

0

(2)

0.01

TABLE IL EARNING PERFORMANCE OF SPIRAL OF 130 POINTS .

(1)
(2)
(3)

(1)

MSE

indexes which are average, minimum, maximum, and standard
deviation. We show the learning performance of the MLPs in
Table I. From this result, the standard MLP (1) is the worstof all, because this method often falls into local minimum.
The proposed MLP (3) improves the performance from theprevious MLP (2). We consider that the unimportant neurons
are removed by the neurogenesis, and the newborn neuron
works on the learning.

In this study, we have proposed the MLP with pulse glial
chain including the neurogenesis. In this network, the glias
are one-by-one connected with the neurons in the hiddenlayer. The glia is excited by the connecting neuron output,
and excited glia generates the pulse. This pulse transmits tothe other glia and increases the threshold of the neuron. The
pulse generation patterns are variedly changed because theperiod of inactivity of the glia is dynamically chain according
to the pulse generation frequency. In addition, the neurogenesis
occurs at half of iterations. By the neurogenesis, some neurons
are removed and the newborn neurons which has random
weights are set in the same position as the removed neurons.
The removed neurons are chosen by the pulse generation
frequency. The neurogenesis increases the number of theimportant neurons. By the computer simulation, we confirmedthat the proposed MLP obtains the better performance thanthe previous MLPs.

The dependency on the number of removed neurons.

ACKNOWLEDGMENTS

C. Dependency on the number of neurons in the hidden-layer
We change the number of neurons in the hidden-layer
and compare the average of error. The dependency of thenumber of neurons is shown in Fig. 4. In general, the learning
performance of the MLP depends on the number of neurons.
However, the average error of the standard MLP does notchange by the number of the neurons. On the other hand,
the MLP with pulse glial chain based on individual period
of inactivity and the proposed MLP decrease the average
error according to increasing of the number of neurons. The
difference of the average error between the MLP with pulse
glial chain based on individual period of inactivity and the
proposed MLP is small in the small number of neurons.
We consider that the proportion of the number of importantneurons is lager than the number of the unimportant neurons
in small number of neurons. Thereby, the neurogenesis is
not efficient to the learning performance in small number ofneurons. When the number of neurons increases, the numberof unimportant neurons also increases. In the large numberof the neurons, the neurogenesis removes many unimportantneurons and set the newborn neurons. Thus, the proposed

This work was partly supported by MEXT/JSPS Grant-inAid for JSPS Fellows (24·10018).
R EFERENCES[1] P.G. Haydon, “Glia: Listening and Talking to the Synapse,” Nature
Reviews Neuroscience, vol. 2, pp. 844-847, 2001.
[2] S. Koizumi, M. Tsuda, Y. Shigemoto-Nogami and K. Inoue, “Dynamic
Inhibition of Excitatory Synaptic Transmission by Astrocyte-Derived
ATP in Hippocampal Cultures,” Proc. National Academy of Science of
U.S.A, vol. 100, pp. 11023-11028, 2003.
[3] S. Ozawa, “Role of Glutamate Transporters in Excitatory Synapses inCerebellar Purkinje Cells,” Brain and Nerve, vol. 59, pp. 669-676, 2007.
[4] G. Perea and A. Araque, “Glial Calcium Signaling and Neuro-Glia
communication,” Cell Calcium, vol. 38, 375-382, 2005.
[5] C. Ikuta, Y. Uwate, and Y. Nishio, “Investigation of Multi-Layer Perceptron with Pulse Glial Chain Based on Individual Inactivity Period,”
Proc. WCCI’14, Jul. 2014 (Accepted).
[6] D.E. Rumelhart, G.E. Hinton and R.J. Williams, “Learning Representations by Back-Propagating Errors,” Nature, vol. 323-9, pp. 533-536,
1986.
[7] H. Song, C.F. Steven and F.H. Gage, “Astroglia Induce Neurogenesis
from Adult Neural Stem Cells,” Nature, pp. 39-44, 2002.
[8] J.R. Alvarez-Sanchez, “Injecting knowledge into the Solution of the
Two-Spiral Problem,” Neural Computing & Applications, vol. 8, pp. 265272, 1999.
[9] H. Sasaki, T. Shiraishi and S. Morishita, “High precision learning forneural networks by dynamic modification of their network structure,”
Dynamics & Design Conference, pp. 411-1–411-6, 2004.

- 72 -