IEEE Workshop on Nonlinear Circuit Networks
December 13-14, 2013

Improvement of Multi-Layer Perceptron by Pulse
Glial Network with Dynamic Period of Inactivity
Chihiro Ikuta

Yoko Uwate

Yoshifumi Nishio

Dept. of Electrical and Electronics Eng., Dept. of Electrical and Electronics Eng., Dept. of Electrical and Electronics Eng.,
Tokushima University
Tokushima University
Tokushima University
2–1 Minami–Josanjima,
2–1 Minami–Josanjima,
2–1 Minami–Josanjima,
Tokushima Japan
Tokushima Japan
Tokushima Japan
Email: ikuta@ee.tokushima-u.ac.jp
Email: uwate@ee.tokushima-u.ac.jp
Email: nishio@ee.tokushima-u.ac.jp

Abstract— In this study, we propose a Multi-Layer Perceptron(MLP) with pulse glial network having dynamic period ofinactivity. We connect glias to neurons in a hidden-layer. The
glia is excited by the connecting neuron output. Then, the glia
generates a pulse. The pulse is propagated to the connecting
neuron and the neighboring glia. In the previous method, we
fix a period of inactivity. The period of inactivity decides thefrequency of the pulse generation. Thereby, the pulse generation
pattern often becomes periodic. We consider that it is similar tothe local minimum. In this network, the period of inactivity of theglia is dynamically changed according to a pulse generation time.
By varying the period of inactivity, the pulse generation patternobtains the diversity. We consider that this diversity of the pulse
generation pattern is efficiency to the MLP performance. By thesimulation, we confirm that the proposed MLP improves theMLP performance than the conventional MLP.

I. I NTRODUCTION
The neuron and the glia make a higher brain function. Theneuron can transmit an electric signals each other. On theother hand, the glia was considered to an accessory cell of theneuron. However, some researchers discovered that the glia
has important functions [1][2]. The glia can transmit signal
by using ions concentrations which are a glutamate acid, anadenosine triphosphoric acid (ATP), calcium (Ca2+ ), and so on[3]. These ions are also used in a gap junction of the neuron.
Thus, we consider that the glia functions closely relate to thebrain’s information processing.
The glia-neural network is important for a detailed investigation of the brain works. However, the brain research ismainly about the neuron. Especially, the application of theglia has not almost investigated. We therefore applied theglia characteristics to a Multi-Layer Perceptron (MLP) for theapplication of the glia. The MLP is a famous artificial neuralnetwork. This network is composed of layers of neurons.
The MLP is generally learned by Back Propagation (BP)algorithm [4]. By this learning, the MLP can be applied toa pattern learning, a data mining, and so on. However, the BP
algorithm has a local minimum problem because this learningalgorithm uses the steepest decent method. The MLP does nothave the connections in the same layer. The neurons connect
to different layer of neurons thus the neurons do not correlate
in the same layer.

- 69 -In this study, we propose the MLP with pulse glial network
which has dynamic period of inactivity. We introduce thedynamic period of inactivity to each glia. If the glia is excitedby the connecting neuron output, the glia cannot be excited
again during the period of inactivity. The previous model
has same time length of the period of inactivity, thereby
the generation pulse pattern becomes the same cycle. In thismethod, the time length of the period of inactivity is varied
to a short when the glia is continuously excited. The glia
which is excited at short interval, obtains different pulse
generation cycle. We consider that the varying the period ofinactivity breaks the periodic pulse generation. The network
learning obtains a diversity. By the computer simulation, weshow that the pulse generation pattern becomes the diversity,
moreover proposed network obtains a better performance thanthe conventional method.
II. P ROPOSED M ETHOD
In this study, we propose the MLP with pulse glial network
having dynamic period of inactivity. We connect the glias tothe neurons in the hidden-layer. The glia makes the different
network from the neural network. Firstly, the glia receives theconnecting neuron output. If it is over the excitation thresholdof the glia, the glia is excited. The excited glia generates thepulse. This pulse can obtain a negative value and a positive
value according to the neuron output. After that, the pulse isinput to the connecting neuron threshold. Moreover, the pulse
influences to the neighboring glias. The neighboring glias arealso excited by this pulse independent from the connecting
neuron output. In the previous method, we fix the period ofinactivity. The period of inactivity decides the cycle of thepulse generation. Then the pulse generation often became theperiodic. We consider that it reduces the possibility of escaping
out from the local minimum. In the proposed method, we vary
the period of inactivity according to the glia excitation. Whenthe same glia is continuously excited by the connecting neuron,the period of inactivity of this glia becomes a short. The glia
obtains the different period of inactivity each other with time.
Thus, this glia exits the periodic pulse generation because theneighboring gila does not finish the period of inactivity when
this glia finishes the period of inactivity.

A. Glia response

C. Updating rule of neuron

The glia has two different states which are the positive
response and the negative response. We define the output
function as the positive response of the glia in Eq. (1).

The neuron has multi-inputs and a single output. We canchange the neuron output by tuning the weights of connections. The standard updating rule of the neuron is defined byEq. (2).
n∑yi (t + 1) = f wij (t)xj (t) − θi (t) ,(2)ψi (t + 1) ={ 1, {(θ < y ∪ ψnii+1,i−1 (t − i ∗ D) = 1)

∩ (τi ≥ θgi )} , (1)j=1γψi (t), else,
where ψ is an output of a glia, i is a position of the glia,
θn is a glia threshold of excitation, y is an output of aconnected neuron, D is a delay time of a glial effect, τ is local
time of the glia during a period of inactivity, θg is a lengthof the period of inactivity, γ is an attenuated parameter. In
the proposed method, the length of the period of inactivity
is varied according to the pulse generation. If the glia iscontinuously excited by the connecting neuron output, thelength of the period of inactivity becomes a short. Moreover,
if the glia is excited by the neighboring glia pulse, the periodof inactivity of this glia returns to original time length of theperiod of inactivity.
B. Pulse propagation
Figure 1 shows an example of the pulse generation and apropagation. In this figure, some glias are excited and thesegenerate the pulses. If the glia receives the large output of theconnecting neuron, this glia generates the positive pulse whichis shown in blue area. If the glia receives the small outputof the connecting neuron, this glia generates the negative
pulse which is shown in red area. After that this pulses
are propagated to the other glias. Both pulse generations are
similar pattern at first. In the case of (a), we can observea small change of the pulse generation pattern. The pulse
generation pattern is converged with time. On the other hand,the pulse generation pattern (b) piecemeal varies from (a).
Moreover, the pulse generation pattern (b) varies for a longtime than (a). From the figure, the proposed network breaks
the periodic pulse generation and makes the diversity.

Time

Position of the neuron from 1st to 40th

where y is an output of the neuron, w is a weight ofconnection, x is an input of the neuron, and θ is a thresholdof neuron. In this equation, the weight of connection and thethreshold of the neuron are learned by BP algorithm. Thus, theneuron output is depended on the BP learning. Next, I show a
proposed updating rule of the neuron. We add the glial effect tothe threshold of neuron. This updating rule is used to neurons
in the hidden layer. It is described by Eq. (3).
n∑wij (t)xj (t) − θi (t) + αψi (t) ,(3)yi (t + 1) = f j=1where α is a weight of the glial effect. We can change theglial effect by change of α. In this equation, the weight ofconnection and the threshold are changed by BP algorithm
same to the standard updating rule of the neuron. However,
the glial effect is not changed. It is updated by Eq. (1).
Equations (2) and (3) are used a sigmoidal function to an
activating function which is described by Eq. (4).
f (a) =1
1 + e−a(4)where a is an inner state.
III. S IMULATIONS
We compare five kinds of the MLPs which are the standard
MLP (1), the MLP with random noise (2), the MLP withpulse glial chain (3), the MLP with pulse glial network (theperiod of inactivity is random) (4), and the MLP with pulseglial network (the period of inactivity is varied according tothe pulse generation) (5). Every MLP has same number ofneurons and layers. The MLP is composed of 2-40-1 neurons.
We obtains the statistic result from 100 trials. Every trial has
different initial conditions. One trial has 50000 iterations. Weuse Mean Square Error (MSE) for the error function. where N
is a number of learning data, T is a target value, and O is anoutput of MLP. We obtain results which are an average error,a minimum error, a maximum error, and a standard deviationof the results. We use a Two-Spiral Problem (TSP) for thesimulation task. The TSP is famous task for the artificial neuralnetwork [5].
A. Statistic results(a) Previous pulse generation (b) Proposed pulse generation

Fig. 1. Pulse generation and propagation. (a) The pulses are generated by
the previous glial network. (b) The pulses are generated by the proposed glialnetwork.

- 70 -Firstly, we show the learning performance of the spirals. Thelearning performance means the fitting between the output ofthe MLP and the supervised classification. The statistic resultshows in Table I. We can see that the standard MLP often
traps into the local minimum. Thereby, the average of error isthe worst of all. The result of the MLP with random noise issimilar to the standard MLP. From this result, the uniformed
random noise is not efficiency to the TSP. Other three MLPs
improve the performance from the result of the standard
MLP. Especially, the MLP with pulse glial chain and theproposed MLP have a good learning performance. Moreover,the maximum error of the proposed MLP is the best of all.
From this result, we can say that the proposed MLP has a highability for escaping out from the local minimum. Thereby, ourproposed MLP reduces an initial valued dependence. It meansthat we can stably obtains the better result.

ACKNOWLEDGMENT

TABLE IL EARNING PERFORMANCE .

(1)
(2)
(3)
(4)
(5)

Average
0.12269
0.108470.019900.05546
0.01414Minimum0.00831
0.000470.000670.001340.00052

Maximum
0.23857
0.24278
0.116640.144810.04851

This work was partly supported by MEXT/JSPS Grant-inAid for JSPS Fellows (24·10018).

Std. Dev.
0.055540.057420.02226
0.03608
0.01313R EFERENCES

B. Learning curves
Fig. 2 shows learning curves of each MLP. The standard
MLP converges the error reduction at 25000. It is trapped
into the local minimum. The convergence of the error inthe MLP with random noise is a slower than the others.
However, it reduces the error than the standard MLP. The
uniformed random noise has a small efficiency to the learningof the MLP. On the others, these curves have a oscillation
during the iterations. Moreover the performance of the error
reduction improves. The pulse locally gives the large energy
to the network. The pulse helps escaping out from the local
minimum. The glia has the period of inactivity. During theperiod of inactivity, the glia does not generate the pulse again.
Thereby, the MLP can search the better solution during theperiod of inactivity. The error reduction of the proposed MLP
is earlier than the others. Thus, the pulse generation patterninfluences the learning of the MLP.
0.35
0.3

MSE

0.25
䠄2䠅

0.2
0.15

䠄1䠅
䠄4䠅

0.1

䠄3䠅
䠄5䠅

0.05
00

10000

20000

30000

40000

50000

Iteration

Fig. 2.

connected with the neuron. The glias generate the pulse according to the connecting neuron output. The generated pulse
influences the connecting neuron threshold and the neighboring glia state. In this method, the period of inactivity is varied
according to the pulse generation time. If the pulse generationcontinuously occurs by the connecting neuron output. By thisinfluence, the pulse generation pattern is dynamically changed
because the period of inactivity of the glia is different eachother. We consider that the glia pulse improves the MLP
performance. Actually, we confirm that the proposed MLP
obtains a better performance than the conventional MLP.

Learning curve.

IV. C ONCLUSIONSIn this study, we have proposed the MLP with pulse glial
network having dynamic period of inactivity. The glias are- 71 -[1] P.G. Haydon, “Glia: Listening and Talking to the Synapse,” NatureReviews Neuroscience, vol. 2, pp. 844-847, 2001.
[2] S. Koizumi, M. Tsuda, Y. Shigemoto-Nogami and K. Inoue, “DynamicInhibition of Excitatory Synaptic Transmission by Astrocyte-Derived
ATP in Hippocampal Cultures,” Proc. National Academy of Science ofU.S.A, vol. 100, pp. 11023-11028, 2003.
[3] S. Ozawa, “Role of Glutamate Transporters in Excitatory Synapses inCerebellar Purkinje Cells,” Brain and Nerve, vol. 59, pp. 669-676, 2007.
[4] D.E. Rumelhart, G.E. Hinton and R.J. Williams, “Learning Representations by Back-Propagating Errors,” Nature, vol. 323-9, pp. 533-536,
1986.
[5] J.R. Alvarez-Sanchez, “Injecting knowledge into the Solution of theTwo-Spiral Problem,” Neural Computing & Applications, vol. 8, pp. 265272, 1999.