IEEE Workshop on Nonlinear Circuit Networks
December 14-15, 2012

Characteristics of Multi-Layer Perceptron Switching
between Learning and Non-Learning Neurons
Chihiro Ikuta

Yoko Uwate

Yoshifumi Nishio

Dept. of Electrical and Electronic Eng., Dept. of Electrical and Electronic Eng., Dept. of Electrical and Electronic Eng.,
Tokushima University
Tokushima University
Tokushima University
2-1 Minami-Josanjima,
2-1 Minami-Josanjima,
2-1 Minami-Josanjima,
Tokushima Japan
Tokushima Japan
Tokushima Japan
Email: ikuta@ee.tokushima-u.ac.jp
Email: uwate@ee.tokushima-u.ac.jp
Email: nishio@ee.tokushima-u.ac.jp

Abstract— A glia has important functions for a higher brain
functions. Among them, we notice that the glia influences thesynaptic Long-Term Potentiation (LTP). The fired glia increase aCa2+ concentration. The Ca2+ generates a d-serine. The d-serine
happens the synaptic LTP. In previous study, we proposed theMulti-Layer Perceptron (MLP) with switching learning and nonlearning neurons. In this method, neurons in the hidden-layer areconnected with glias and separated to some groups according tothe connected glia. The glias are regularly fired. The neurons
are switched the learning and non-learning terms according to
the glia firing. We consider that switching between two terms isefficient to the MLP performance. By the simulations, we confirm
that the change of the number of neurons is how to influence tothe learning performance.

standard MLP is often falls into local minimum, because theBP uses the steepest decent method. The proposed MLP haslearning neurons and non-learning neurons during iterations.
We consider that the non-learning neurons gives an energy to
the MLP for escaping out from the local minimum. The nonlearning neurons periodically change to the learning neurons
with time. The hidden-layer neurons repeat the switching of
the learning term and the non-learning term. By the simulations, we confirm the performance and characteristics of theproposed MLP.

I. I NTRODUCTION

II. P ROPOSED M ETHOD

Glia had not been investigated, because it was considered
to the support cell of a neuron. However, some researchers
discovered that the glia transmits signal by using various ions
[1][2]. For example, the ions are an adenosine triphosphate,a glutamate acid, a calcium, and so on [3][4]. They areimportant for the brain works. Actually, they are often usedin the synapse. Among them, we have noticed the Ca2+ .
Takatas reported that the Ca2+ affects the synaptic Long-TermPotentiation (LTP) by an experiment on living animals [5]. In
this research, the increase of the Ca2+ concentration induces
the d-serine at the periphery of the fired synapse. The increaseof the Ca2+ concentration is generated by the glia. If the glia
does not increase the Ca2+ , the d-serine is not generated. Thed-serine relates closely the LTP, thus the LTP is not observed
when the increase of the Ca2+ is not happened.
In previous study, we proposed a Multi-Layer Perceptron(MLP) with switching between learning and non-learningneurons which is presented in NOLTA’12 [6]. In the biologicalsystem, the neurons are learned in the increase of the Ca2+
concentration, moreover the glial effect is propagated. We
attract these features and apply these features to the MLP.
W connected the glias with the neurons in the hidde-layer
and the neurons are separated to some groups. The glias areregularly fired. The neurons in a same group learn at sametime. Thus the neurons are switched between learning andnon-learning term according to the firing of glias. The MLP
learns by a Back Propagation (BP) algorithm [7]. However, theThe MLP is a famous feed forward neural network. This
network is composed of layers of neurons and the outputs aretuned by changing the weights of connections. In general, it
is learned by the BP which was proposed by D.E. Rumelhart[7]. The MLP can be applied to a pattern recognition, a patternclassification, a data mining, and so on. However, the MLP has
the local minimum problem, because the BP uses the steepest
decent method. For the local minimum problem, we often add
a noise to any part of the MLP.
The MLP with switching between learning and non-learningneurons was proposed in NOLTA’12. The proposed MLP isinspired from relationships the neurons and the glias in the
biological system. The glia is one of nervous cells in thebrain. This cell can change the Ca2+ concentration which ispropagated to neighboring glias [8][9]. The Ca2+ is important
for the LTP of the synapse. In order to increase the response
amplitudes of the synapse by the learning, the glia must
increases the Ca2+ concentration [5]. We apply these features
to the MLP. In the proposed MLP, the hidden-layer neurons
are separated some learning groups. The neurons in the samegroup learn by the BP algorithm at same time. The groups
periodically switch between the learning term and the nonlearning term. During the learning term, the hidden-layer
neurons are learned. During the non-learning term the weightsof connections between the hidden-layer and the output layer
are not updated.

-1-III. S IMULATIONS

A. Updating rule of neuron
The neuron has multi-inputs and a single output. We canchange the neuron output by tuning weights of connections.
The updating rule of the neuron is defined by Eq. (1).
n∑yi (t + 1) = f wij (t)xj (t) − θi (t) ,(1)j=1

In this simulation, we use the Two-Spiral Problem (TSP)
for a learning task. The TSP is a famous task for the artificial
neural network and it has a high nonlinearity. In this task,
the coordinates of the spirals are input to the MLP and theMLP learns the correlating classifications. The input spirals
are shown as Fig. 2.
1

where y is an output of the neuron, w is a weight ofconnection, x is an input of the neuron, and θ is a thresholdof neuron. In this equation, the weights of the connections
and the thresholds of neurons are learned by BP algorithm.
We used sigmoidal function to an activating function which isdescribed by Eq. (2).
f (a)

=1
1 + e−a0.80.6
y00.4

10.2(2)

000.2

0.4

0.6

0.8

1x

where a is an inner state.
Fig. 2.

Two-Spiral Problem.

B. Flow of the proposed MLP
We show the flow of the learning of the proposed MLP inFig. 1. In this example, one group is composed of two neurons.
Every group has same time length of the learning term andthe non-learning term. The learning term and the non-learningterm are changed with time. During the learning term, theneurons are learned by BP algorithm. On the other hand, theweights of connections between the hidden-layer neuron andthe output-layer neuron are not updated. First group goes into
the learning term. Other groups are the non-learning term.
Second group goes into the learning term, then the first groupremain the learning term. In the middle of the learning termof the second group, the first group finishes the learning termand start the non-learning term. After that, every group repeatschanging the learning term and the non-learning term.
Group 1

The average of error is obtained from 100 trials and eachtrial has 100000 iterations. Moreover, we fix the delay of thelearning term to 5. We use a Mean Square Error (MSE) to anerror function. The MSE is described by Eq. (3).
1 ∑(Tn − On )2 ,N n=1
N

M SE =(3)A. Approximation performance
We compare approximation performances the proposed
MLP and the standard MLP. The simulation result is shown
Table I. From this table, the proposed MLP has better performance than the standard MLP. The TSP is difficult task,
thus the standard MLP is trapped into local minimum. Fromcomparison result, the proposed MLP can escape out fromthe local minimum. Moreover, the proposed MLP has a highsolution stability.
TABLE IDelay

L EARNING PERFORMANCE .
Standard
Proposed

Average
0.10300.0358Minimum
0.00100.0002

Maximum
0.22370.1694

Std. Dev.
0.05490.0322

B. Convergence performance
We show the convergence performance of the proposed
MLP shown as Fig. 3. The decrease of error is fixed about
100000 iterations in the both MLPs. However, in the proposed
MLP, the error is rapidly decreased from 50000 iterations
to 100000 iterations. We can see that the influence of theproposed method is slowly started. Because the proposed MLP
has the non-learning term. During the non-learning term, theweight updating is skipped, thus the decrease of the error isslow.

The number of neurons

Fig. 1.

Flow of the learning of the proposed MLP.

-2-0.2
0.18

20Standard MLP

0.16

40

Proposed MLP

0.14

60

0.12

80

MSEMSE

1

0.1

0.1
0.08
0.06
0.04
0.02

0.01
10000

0100000

51000000

Fig. 3.

10 15 20 25 30 35 40 45 50

Time length of the learning term

IterationFig. 5. The performance of different number of neurons (5 neurons in thesame group).

The convergence performance of the proposed MLP.

C. Number of neurons in the hidden-layer
Figure 1 shows the performances of the proposed MLP
when we change the number of neurons in the hidden-layer.
The proposed MLP which has 40 neurons is the best of all.
In general, the MLP obtains a better performance if we use
the larger number of neurons, however, the MLP which haslarge number on neurons needs a high simulation cost and alonger iterations. In this case, the proposed MLP which has 80
neurons is the worst performance of all. We consider that theproposed MLP which has 80 neurons cannot converge, thus it
obtains the worst result.
0.22040
6080

0.18
0.16
0.14

0.2
0.18

0.12

0.16

0.1

0.14

0.08

0.12

MSEMSE

learning group. We confirmed that all MLP can obtain thesimilar performances to Fig. 5 if they can converge. In thissimulation, the proposed MLPs which have 60 and 80 neurons
improves the performances than Fig. 4. On the other hand,the performance of proposed MLP which has 40 neurons
worth than Fig. 4 when we use the large time length of thelearning term. We consider that this MLP becomes similar tothe standard MLP, because we increase the number of neurons
in the same group and the time length of the learning term.
Thus, the proposed MLP which has 40 neurons is trapped into
local minimum similar to the standard MLP.

0.06

406080

0.1

0.04

0.08

0.02

0.06
0.04

0510 15 20 25 30 35 40 45 50

0.02Time length of the learning term

0Fig. 4. The performance of different number of neurons (5 neurons in thesame group).

We increase the iterations to 1000000 and obtain the result.
Figure 5 shows the simulation results. In this case, every result
is similar to each other. From this result, the proposed MLP
needs the longer iterations when we use the large number ofneurons. Moreover the proposed MLP which has 40 neurons
can balance an approximation performance with a convergenceperformance.
Next, we change the number of neurons in the same group.
In this simulation, we use 100000 iterations for the MLPs.
Figure 6 simulations result when we use 10 neurons to one-3-5

10 15 20 25 30 35 40 45 50Time length of the learning term
Fig. 6. The performance of different number of neurons (10 neurons in thesame group).

In addition, we increase the number of neurons in the samelearning group to 20. The simulation results are shown as
Fig. 7. We can see that the performances improve as the MLP
having the large number of neurons. Moreover, we can saythat the proposed MLP has a negative correlation between thenumber of neurons in the same group and the time length ofthe learning term.

MSE

0.2
0.18

400.1660

0.1480[10] J.R. Alvarez-Sanchez, “Injecting knowledge into the Solution of theTwo-Spiral Problem,” Neural Computing & Applications, vol. 8, pp. 265272, 1999.
[11] H. Sasaki, T. Shiraishi and S. Morishita, “High precision learning forneural networks by dynamic modification of their network structure,”
Dynamics & Design Conference, pp. 411-1–411-6, 2004.

0.12
0.1
0.08
0.06
0.04
0.02
05

10 15 20 25 30 35 40 45 50

Time length of the learning term
Fig. 7. The performance of different number of neurons (10 neurons in thesame group).

IV. C ONCLUSIONSIn this study, we have investigated about the MLP withswitching between learning and non-learning neurons whichis inspired from the biological characteristics of the glia. In
this model, the hidden-layer neurons are connected with theglias and are separated to some groups. The neurons in thesame group is learned at same time. The glias are regularlyfired. Every learning group is switched between learning andnon-learning term according to the firing of glias. By thesimulation, we showed that the proposed method is efficient tothe MLP learning. Moreover, we confirmed the characteristicof the proposed MLP.
ACKNOWLEDGMENT
This work was partly supported by MEXT/JSPS Grant-inAid for JSPS Fellows (24·10018).
R EFERENCES[1] P.G. Haydon, “Glia: Listening and Talking to the Synapse,” Nature
Reviews Neuroscience, vol. 2, pp. 844-847, 2001.
[2] S. Koizumi, M. Tsuda, Y. Shigemoto-Nogami and K. Inoue, “DynamicInhibition of Excitatory Synaptic Transmission by Astrocyte-Derived
ATP in Hippocampal Cultures,” Proc. National Academy of Science ofU.S.A, vol. 100, pp. 11023-11028, Mar. 2003.
[3] S. Ozawa, “Role of Glutamate Transporters in Excitatory Synapses inCerebellar Purkinje Cells,” Brain and Nerve, vol. 59, pp. 669-676, 2007.
[4] G. Perea and A. Araque, “Glial Calcium Signaling and Neuro-Glia
Communication,” Cell Calcium, vol. 38, pp. 375-382, 2005.
[5] N. Takata, T. Mishima, C. Hisatsune, T. Nagai, E. Ebisui, K. Mikoshiba,
and H. Hirase, “Astrocyte Calcium Signaling Transforms Cholinergic
Modulation to Cortical Plasticity In Vivo,” The Journal of Neurocience,
vol. 31, pp. 18155-18165, Dec. 2011.
[6] C. Ikuta, Y. Uwate, Y. Nishio, and Guoan Yang, “Multi-Layer Perceptron
Decided Learning Neurons by Regular Output Glias,” Proc. NOLTA’12,
pp. 719-722, Oct. 2012.
[7] D.E. Rumelhart, G.E. Hinton and R.J. Williams, “Learning Representations by Back-Propagating Errors,” Nature, vol. 323-9, pp. 533-536,
1986.
[8] S. Kriegler and S.Y. Chiu, “Calcium Signaling of Glial Cells along
Mammalian Axons,” The Journal of Neuroscience, vol. 13, pp. 42294245, 1993.
[9] M.P. Mattoson and S.L. Chan, “Neuronal and Glial Calcium Signalingin Alzheimer’s Disease,” Cell Calcium, vol. 34, pp. 385-397, 2003.

-4-