Proceedings of International Joint Conference on Neural Networks, Dallas, Texas, USA, August 4-9, 2013

Investigation of Four-Layer Multi-Layer Perceptron
with Glia Connections of Hidden-Layer Neurons
Chihiro Ikuta, Yoko Uwate, and Yoshifumi Nishio

Abstract— A glia is a nervous cell existing in a brain. This
cell transmits signals by various ion concentrations. By the ion
concentration, the glia correlates the neuron by ions. Thus,
the glia composes the network different to the neural network.
We proposed a four-layer Multi-Layer Perceptron (MLP) withglia connections of hidden-layer neurons which is inspired fromcharacteristics of the biological glia. The proposed MLP has two
hidden-layer. In this network, we connect the glia to the neurons
in two hidden-layer. The glia receives the output of connecting
neuron and the outputs are summed. When the summed value
is over the glia excitation threshold, this glia is excited. The
excited glia generates a pulse. This pulse input to the thresholds
of connecting glias. We consider that the glia gives the energy to
the MLP and the glia give the position relationship of neurons
in the different layers. By the simulation, we confirm that the
proposed MLP obtains a high solving ability. Moreover, weinvestigate the characteristics of the proposed MLP.

I. I NTRODUCTION

B

RAIN has two kinds of nervous cells which are aneuron and a glia. The neuron transmits electric signal
each other. This work was found in an earlier research
stage of the neuron. Thereby, the neuron has been investigated about biological characteristics and applications. On
the other hand, the glia had not been investigated about
details, because this cell was considered to a supportive
cell of the neuron. Recently, some researchers discovered
novel biological characteristics of the glia [1][2]. The novel
biological characteristics that the glia can transmit signals bychanging of ion concentrations such as a Ca2+ , an adenosine
triphosphate, a glutamate acid, and so on [3]. Moreover, thiscell has various receptors for these ions. Currently, the glia
is considered that the glia influences neuron, thus the glia
closely relates to the brain function. In these ions, we notice
the Ca2+ . The glia generates the Ca2+ concentration wave
according to the neuron excitation [4]. The Ca2+ propagatesto a wide range in the brain and influence a membranepotential of neuron [5][6]. Thus, the glia composes the
network different to the neural network. The neurons locally
influence each other. The glias have wider relationships andcompose the wide network than the neuron. We consider
that the glia function can be applied to the artificial neuralnetwork.
In previous studies, we proposed some neuron-glia network [7][8]. We apply the characteristics of the biologicalglia to the Multi-Layer Perceptron (MLP). Because the neurons in the same layer are do not have a position relationship.
Chihiro Ikuta, Yoko Uwate, and Yoshifumi Nishio are with Department ofElectrical and Electronics Engineering, Tokushima University, Japan (email:{ikuta, uwate, nishio}@ee.tokushima-u.ac.jp).

978-1-4673-6129-3/13/$31.00 ©2013 IEEE

In a biological system, the neuron composed of a cluster.
We consider that the position relationship of the neurons
is important for the artificial neural network. We connect
the glia to the neurons in the hidden-layer. The glia andthe neighboring glia influence each other. Moreover, the glia
gives the position relationships to the connecting neurons.
In this study, we propose a four-layer MLP with glia
connections of hidden-layer neurons. We give the glias tothe hidden-layer neuron. In this model, we use the fourlayer MLP, thus the MLP has two hidden-layer. The first
hidden-layer neurons connects with every second hiddenlayer neuron. The relationship of the neurons is equivalencein each position. We connect the glia to the hidden-layer
neurons. In this model, the glia is connected with someneurons which exists in first hidden-layer and second hiddenlayer. The glia receives the output of each neuron and these
outputs are summed. When the sum of these outputs isover the excitation threshold of glia, the glia is excited.
The excited glia generates the pulse. The pulse is input tothe connecting neuron threshold. Thus, the neuron shares
the same pulse. In a standard MLP, the neurons do nothave the position relationships because these connections areequivalence. By our method, the neighboring neurons areconnected by the glia pulse. The neurons obtain the position
relationships of neurons in the hidden-layer. By the computer
simulations, we show that the proposed model has a highsolving performance than the standard model.
II. P ROPOSED METHOD
The MLP is one of feed forward neural networks. It can beapplied to a data mining, a pattern classification, a patternrecognition, and so on. In this model, a solution accuracy
is depended on the number of neurons and a detention of
nonlinear separation is depended on the number of layers.
We can tune the output of this network by changing weightof connections between neuron. We often use a Back Propagation (BP) algorithm for decision of weights of connections.
This algorithm is proposed by Rumelhart in 1986 [9]. In the
BP algorithm, the error of network output is propagated to thebackward. Then, the error decreases. However, this algorithm
is often falls into the local minimum, because it is used to thesteepest decent method. In general, the energy is given to the
network for escaping out from local minimum. The energy
helps for the escaping out from the local minimum. Theenergy is efficiency to the local minimum problem, however
the learning of network becomes oscillatory. Moreover, theadditional energy algorithm is difficult and it often needs alarge computational cost.

1523

In this study, we propose the four-layer MLP with glia
connections of hidden-layer neurons. The proposed network
is shown in Fig. 1. We connect the glia to the hidden-layer
neurons in a cross different layers. The glia receives every
connecting neuron output and these outputs are summed.
When the sum of the neuron output is over the excitationthreshold of the glia, the glia is excited. The excited glia
generates the pulse which becomes positive response to thesum of neuron output. After that, the pulse is input to theneuron threshold because the glia influences the membranepotential of neuron in the biological system. The connecting
neurons receive the same pulse at same time, thus theseneurons work become similar on a short-term basis. We
consider that the glia composes the local group of theneurons. In the standard model, the network learns at thesame time. In our method, the network learns at different
time in each group.

Eqs. (1) and (2) are used sigmoidal function to activating
function which is described by Eq. (3).
𝑓 (𝑎)

=

1
1 + 𝑒−𝑎(3)where 𝑎 is an inner state.
B. Back propagation algorithm
The error of MLP propagates backward in the network.
The BP algorithm changes value of weights to obtain smaller
error than before. The error of the network is given by
Eq. (4).
𝑛𝐸=

1∑(𝑡𝑖 − 𝑂𝑖 ),
2 𝑖=1

(4)

where 𝐸 is error value, 𝑡 target value and 𝑂 is the outputof network. The error of network decrease by changing
weights of connections according to the partial differentiationof error. Therefore, partial differentiation of the error isdescribed by Eq. (5).
∂𝐸
,
(5)
∂𝑤
where Δ𝑊 is an updated weight and 𝜂 is a learning
coefficient.
Δ𝑊 = −𝜂C. GliaFig. 1.

Proposed MLP.

A. Updating rule of neuron
The neuron is multi-input and single output. The neuron
receives the outputs of neurons in forward layer. The standard
updating rule of neuron is described by Eq. (1).
⎛⎞𝑛∑𝑤𝑖𝑗 (𝑡)𝑥𝑗 (𝑡) − 𝜃𝑖 (𝑡)⎠ ,(1)𝑦𝑖 (𝑡 + 1) = 𝑓 ⎝𝑗=1

where 𝑦 is an output of the neuron, 𝑤 is a weight ofconnection, 𝑥 is an input of the neuron, and 𝜃 is a thresholdof neuron. In this equation, 𝑤 and 𝜃 are tuned by the
BP algorithm. Thus, if the network is trapped into local
minimum, it cannot escape from there. Next, I show a
proposed updating rule of the neuron. We add the glial effectto the threshold of neuron. This updating rule is used toneurons in the hidden-layer. It is described by Eq. (2).
⎛⎞𝑛∑𝑤𝑖𝑗 (𝑡)𝑥𝑗 (𝑡) − 𝜃𝑖 (𝑡) + 𝛼𝜓𝑖 (𝑡)⎠ , (2)𝑦𝑖 (𝑡 + 1) = 𝑓 ⎝𝑗=1

where 𝛼 is a weight of glial effect and 𝜓 is pulse of glia. We
can control the glial effect by change of 𝛼. In this equation,
the weight of connection and the threshold are learned by BP
algorithm as same as the standard updating rule of neuron.

The glia is the nervous cell existing in the brain. This cell
was considered to a static cell because we could not observed
the work of glia. Recently, some researchers discovered thatthe glia transmits signals by ions concentrations according
to the development observation technology. The glia uses
various ions which are Ca2+ , glutamate acid, GABA, andso on. Among them, we notice Ca2+ . The glia is excited
according to the neuron firing. The excited glia generates
the Ca2+ concentration wave. The Ca2+ concentration wave
transmit to the neighboring glias and neurons. The glia
composes the network different to the neural network. We
have considered that the glia can be applied to the artificial
neural network.
In this study, we connect the glia to the hidden-layer
neurons. We use four-layer MLP, thus this network has
two hidden-layer. The glia is connected to the neurons in
two hidden-layer. The glia receives the connecting neurons
outputs. These outputs are summed and held according to
Eq. (6).
𝑔=𝑏
1 ∑(𝐻1𝑖 + 𝐻2𝑖 − 1.0),𝑏 − 𝑎 𝑖=𝑎(6)where 𝑔 is a held output value, 𝑏 − 𝑎 is the number ofconnecting neurons in the one layer, 𝐻1𝑖 is a neuron output
in the first hidden-layer, and 𝐻2𝑖 is a neuron output in thesecond hidden-layer. When the held output value is over thethreshold of excitation glia, the glia is excited. The excitedglia generates the pulse which becomes the positive response.
The pulse inputs to the connecting neuron threshold, thus, theconnecting neurons share the same threshold change. The

1524

glia has the period of inactivity. Although the held output
value of glia is over the excitation threshold of glia, the glia
cannot generate the pulse during the period of inactivity. The
glia response is described by Eq. (7).
{ 1, 𝐼 = 0, (𝑔 > 𝜃 ∩ 𝐼 > 𝜃 )𝜓𝑖 (𝑡 + 1) =𝑔𝑡

−1, 𝐼 = 0, (𝑔 < −𝜃𝑔 ∩ 𝐼 > 𝜃𝑡 ) ,𝛾𝜓𝑖 (𝑡), 𝐼 = 𝐼 + 1, 𝑒𝑙𝑠𝑒,(7)

where 𝜓 is a glia output, 𝛾 is an attenuated parameter, 𝜃𝑔
is an excitation threshold of the glia, 𝐼 is a local time of
the glia, and 𝜃𝑡 is a time length of a period of inactivity. In
our method, the positive and negative pulses are generated
according to 𝑔. The glia output is decreased in an exponential
fashion. The glia response is decided by the parameters andthe parameters fix during the iterations. However, the pulse isgenerated according to the neuron output. The neuron output
is changed by learning. Thereby, the generation pattern ofpulse is dynamically changed according to the learning. We
show an example of the pulse generation in Fig. 2. From thisfigure, the neuron output is changed with time. The neuron
output becomes zero when it excites the glia. We can see thatthe pulse is generated according to the neuron output, andthe pulse generation density is changed. Because the neuron
output is changed by learning. Moreover, the pulse valuechange from 1 to −1. Every glia changes the pulse generation
pattern similar to Fig. 2 during the iteration. The neuron andglia correlate each other. We consider that the characteristic
pulse generation pattern is efficiency to the MLP learning.

(3)

The four-layers MLP with glia connection of
hidden-layer (The relationships between the neuron
and glia in the same layer).
(4)
The four-layer MLP with glia connection of hiddenlayer.
The standard MLP does not have the external unit. Thismethod early converges the error, however it is often falls
into local minimum. The MLP with random noise is input
the uniformed random noise to every neuron in the hiddenlayer. (3) and (4) uses the proposed method. In the case of(3), the glia is connected with the neuron in the same layer,
thus the glia only receives neuron output from same layer
and the generated pulse inputs to the neuron in the samelayer. The glia which is connected with neurons in the first
hidden-layer and the glia which is connected with neurons
in the second hidden-layer are independent each other. (4)has the connection between first hidden-layer and second
hidden-layer, thereby the neurons which connects the same
glia receive the same pulse at the same time. Each MLP is
composed of 4 layers. The input-layer has 2 neurons. The fist
hidden-layer has 10 neurons. The second hidden-layer has
10 neurons. The output-layer has 1 neuron. In the proposed
MLP, the glia is connected with two neurons in one hiddenlayer. Thus, one glia is connected with 4 neurons.
We evaluate the result by a Mean Square Error (MSE)
which is shown as Eq. (8).
𝑁

𝑀 𝑆𝐸 =

1

ψ(t)

(8)

where 𝑁 is a number of learning data, 𝑇 is a target value,
and 𝑂 is an output of MLP. We give 100 trials to each MLP
and one trial has 500000 iterations. Every MLP has same
initial conditions. From 100 trials, we obtain four kinds ofindexes of MSE which are an average of error, a minimum
error, a maximum error, and a standard deviation of error.

0.5
0

-0.5
-1
0

10000

20000

0

10000

20000

30000

40000

50000

30000

40000

50000

60
40
20

g(t)

1 ∑
(𝑇𝑛 − 𝑂𝑛 )2 ,
𝑁 𝑛=1

0
-20
-40
-60

Fig. 2.

t

An example of pulse generation.

III. S IMULATIONIn this section, we show the simulation result. We use 4kinds of the MLPs for comparison of the performance.
(1)
The standard MLP.
(2)
The MLP with random noise.

A. Simulation task
We use the Two-Spiral Problem (TSP) for a comparisonof performance. The TSP is a famous task for the artificial
neural network [10][11]. This task is linearly-inseparable
problem, moreover this task has a high nonlinearity. In
this task, the MLP is input the coordinates of spiral point
and learns the classification of this point. The MLP learns
every classification of spiral points. The learning spiral points
shows Fig. 3 (a). Figure 3 (b) is the ideal result of TSP. It is
calculated by norm between spiral points and coordinates.
B. Learning performanceFirst, we show the statistic learning performance of the
MLPs. The results shows in Table I. The error averageof standard MLP is the worst of all. Because this method
often traps into the local minimum. The learning of thestandard MLP is early converged. The MLP with random
noise can find the best result of all. However, the error
average is worth than the proposed MLP. In this model, every
neuron in the hidden-layer received the uniformed random

1525

1

1

0.8

0.8

0.6

Standard
Random noise
Proposed

0.6
0

y

y

0
1

0.4

1

1

0.4

0.1
0.2

0
0

Fig. 3.
TSP.

0.2

0.4

x
(a)

0.6

0.8

1

0
0

0.2

0.4

0.6

x
(b)

0.8

1

Two-spiral problem. (a) Learning spiral points. (b) Ideal result of

noise. The doping noises does not have a correlations. We
consider that the learning of this MLP is oscillatory. The
two proposed model which receive the pulse from the glia.
These performances are different. From this table, we cansay that the position relationships of pulses are important for
the MLP learning performance.

MSE

0.2

0.01

0.001
0.E+00 1.E+05 2.E+05 3.E+05 4.E+05 5.E+05

Iteration
Fig. 4.

TABLE IL EARNING PERFORMANCE .
(1)
(2)
(3)
(4)

Average0.01384
0.012200.01037
0.00905

Minimum
0.00001
0.00000
0.000020.00001

Maximum
0.08687
0.094700.078900.07259

Examples of learning curves.

TABLE IIC LASSIFICATION PERFORMANCE .
Std. Dev.
0.01959
0.01772
0.01427
0.01234

We show an example of learning curve shown as Fig. 4.
The learning curve of standard MLP converges earlier. However, it stops the decreasing error. The curve of MLP withrandom noise has small oscillations. Every neuron in thehidden-layer receives the random noise. Thus, the osculation
is observed over all iterations. The proposed MLP has rapidchange of error. We consider that the MLP receives the pulse
at same time. The works of glias are independent each other.
However, the pulses are stochastically generated at sametime. Then, the MLP obtain a large energy from the glia,
after that the MLP escapes out from the local minimum.
C. Classification performance
Next, we compare the generalization capability of the
MLPs. The coordinates between 0 and 1 are input to thelearned MLPs. We compare the ideal classification result(Fig. 3 (b) with output of the network. The result show inTable II. In this result, the MLP with random noise is thebest of all. In the classification, the MLP needs the wide
area search ability. The neurons in the hidden-layer has thedecorrelation noise. Thus, the MLP with random noise cansearch the wide area solutions. The performance of proposed
MLP is worth than the MLP with random noise. The neurons
receive the same pulse, thus the learning of the neurons in
the hidden-layer is low variability.
Figure 5 is the classification image by each MLP. We use
the best results of each MLP in Table II. The standard MLP
has some deficit spaces. The MLP with random noise and

(1)
(2)
(3)
(4)

Average
0.146290.129150.139630.13422

Minimum
0.09879
0.08278
0.095780.08517

Maximum
0.225810.182200.20769
0.20657

Std. Dev.
0.02469
0.02132
0.02576
0.02208

the proposed MLP (The relationships between the neuron
and glia in the same layer) have one deficit space. In the
MLP with random noise has a obvious error in a bottom
part of image. The proposed MLP does not have the deficit
space different from others. From these images, we can saythat the MLP with random noise has a high performance inlocal part of image. On the other hand, the proposed MLP
can classifier to two spiral in the wide space.
D. Parameter characteristics
Finally, we show the parameter characteristics of the
proposed MLP. We change the parameters of the glia. The
horizontal line means weight of glial effect (𝛼). The verticalline means the MSE. We change 𝛾 from 0.7 to 0.95. Eachgraph shows the result by different attenuated parameter (𝛾).
If 𝛾 is higher, the decreasing of the pulse becomes slow.
Figures 6-8 have different 𝜃𝑡 . We cannot observe the highrelationships of weight of glial effect and attenuated parameter to the learning performance. In the case of classification
performance, it has a high dependency to the weight of glial
effect and attenuated parameter to the learning performance.
In Fig. 6, the error of learning performance increases for
𝛾 = 0.95. 𝜃𝑡 = 30 means that the glia can rapidly finish
the period of inactivity, thus the new pulse is generated
when the pulse is not enough to decrease. The classification
performance improves as increasing 𝛼. When 𝛼 is over 0.3,
the performance is decreased. However, the error is large

1526

1

0.025

0.8

0.8

0.02

0.16

0.6

0.015

0.15

0.6
0.4

0.4

0.2

0.2

0.17

0.95
0.13

0.005
0

0.12
0.2

(1)

0.6

0.8

1

0

1

1

0.8

0.8

0.6
0.4
0.2

0.4

0.6

0

0.2

0.2

0.4

(2)

0.6

0.8

α

0.4

0.6

(b)

1

Fig. 7. Parameter characteristics between weight of glial effect (𝛼) andthe MSE (𝛾 = 0.7, 0.8, 0.9, 0.95, 𝜃𝑡 = 40). (a) Learning performance. (b)Classification performance.
0.025

0.17

0.6

0.02

0.16

0.4

0.015

0.15

0.7

MSE

0.4

MSE

0.2

α

(a)

0
0

0.8
0.9

0.14

0.01

0

0

0.7

MSE

MSE

1

0.01

0.8
0.9

0.14

0.2

0.95
0.005

0

0
0

0.2

0.4

(3)

0.6

0.8

1

0

0.2

0.4

(4)

0.6

0.8

1

0.13

0
0

0.2

α

0.4

0.6

0.12
0

0.2

(a)

Fig. 5.

α

0.4

0.6

(b)

Classification images.

when 𝛾 = 0.95. From two results, the new pulse should be
generated when the pulse is enough decreased.

Fig. 8. Parameter characteristics between weight of glial effect (𝛼) andthe MSE (𝛾 = 0.7, 0.8, 0.9, 0.95, 𝜃𝑡 = 50). (a) Learning performance. (b)Classification performance.

IV. C ONCLUSIONS
0.06

0.17

0.05

0.16

MSE

MSE

0.04
0.03
0.02

0.7

0.15

0.8
0.9

0.14

0.95
0.13

0.01
0
0

0.2

α

(a)

0.4

0.6

0.12
0

0.2

α

0.4

0.6

(b)

Fig. 6. Parameter characteristics between weight of glial effect (𝛼) andthe MSE (𝛾 = 0.7, 0.8, 0.9, 0.95, 𝜃𝑡 = 30). (a) Learning performance. (b)Classification performance.

Figure 7 shows the result for 𝜃𝑡 = 40. The error oflearning is increased for comparison with Fig. 6. We consider
that the 𝜃𝑡 influence to the learning performance. In thiscase, dependency for 𝛼 and 𝛾 is smaller than the previous
result. From this result, we consider that the repetition ofpulse is important for the MLP learning performance. The
classification performance improves than the previous resultin many conditions.
Figure 8 is the case of 𝜃𝑡 = 50. We cannot observe
the dependency to 𝛼 and 𝛾 in the learning performance.
However, the classification performance is similar to Fig. 7(b). From these results, the learning performance has high 𝜃𝑡dependency and the classification performance has a high 𝛼
and 𝛾 dependency.

In this study, we have proposed four-layer MLP withglia connections of hidden-layer neurons. In this method,
the MLP is composed of four neuron layers. We connect
the glia to the neurons in two hidden-layer. The glia gives
the relationships between the neurons in the first hiddenlayer and the neurons in the second-hidden layer. The glia
receives the outputs of connecting neurons. These outputs
are summed and held by the glia. The glia is excited by
the summed value of neuron output when this value isover the excitation threshold. The excited glia generates
the pulse and this pulse is input to the connecting neurons
thresholds. Thus, the connecting neurons have same pulse
at same time. We consider that the glia pulse give theposition relationships of the neurons in the hidden-layer. By
solving the TSP, we confirmed that the proposed MLP has
better performance than the standard MLP. Moreover, weinvestigated the parameter characteristics. We confirmed thatthe learning performance and the classification performanceof the proposed MLP are depended to different parameters.
ACKNOWLEDGMENT
This work was partly supported by MEXT/JSPS Grant-inAid for JSPS Fellows (24⋅10018).
R EFERENCES[1] P.G. Haydon, “Glia: Listening and Talking to the Synapse,” Nature
Reviews Neuroscience, vol. 2, pp. 844-847, 2001.

1527

[2] S. Koizumi, M. Tsuda, Y. Shigemoto-Nogami and K. Inoue, “Dynamic
Inhibition of Excitatory Synaptic Transmission by Astrocyte-Derived
ATP in Hippocampal Cultures,” Proc. National Academy of Scienceof U.S.A, vol. 100, pp. 11023-11028, Mar. 2003.
[3] S. Kriegler and S.Y. Chiu, “Calcium Signaling of Glial Cells along
Mammalian Axons,” The Journal of Neuroscience, vol. 13, pp. 42294245, 1993.
[4] S. Ozawa, “Role of Glutamate Transporters in Excitatory Synapses
in Cerebellar Purkinje Cells,” Brain and Nerve, vol. 59, pp. 669-676,
2007.
[5] M.P. Mattoson and S.L. Chan, “Neuronal and Glial Calcium Signalingin Alzheimer’s Disease,” Cell Calcium, vol. 34, pp. 385-397, 2003.
[6] G. Perea and A. Araque, “Glial Calcium Signaling and Neuro-Glia
Communication,” Cell Calcium, vol. 38, pp. 375-382, 2005.
[7] C. Ikuta, Y. Uwate, and Y.Nishio, “Multi-Layer Perceptron with
Positive and Negative Pulse Glial Chain for Solving Two-Spirals
Problem,” Proc. IJCNN’12, pp. 2590-2595, Jun. 2012.
[8] C. Ikuta, Y. Uwate, Y. Nishio, and G. Yang, “Multi-Layer Perceptron Decided Learning Neurons by Regular Output Glias,” Proc.
NOLTA’12, pp. 719-722, Oct. 2012.
[9] D.E. Rumelhart, G.E. Hinton and R.J. Williams, “Learning Representations by Back-Propagating Errors,” Nature, vol. 323-9, pp. 533-536,
1986.
[10] J.R. Alvarez-Sanchez, “Injecting knowledge into the Solution of the
Two-Spiral Problem,” Neural Computing & Applications, vol. 8,
pp. 265-272, 1999.
[11] H. Sasaki, T. Shiraishi and S. Morishita, “High precision learning forneural networks by dynamic modification of their network structure,”
Dynamics & Design Conference, pp. 411-1–411-6, 2004.

1528