2009 IEEE Workshop on
Nonlinear Circuit Networks, Tokushima, Dec 11-12, 2009

Multi-Layer Perceptron with Chaos Glial Network
Chihiro Ikuta

Yoko Uwate

Yoshifumi Nishio

Dept. of Electrical and Electronic Eng.,
Tokushima University
Email: ikuta@ee.tokushima-u.ac.jp

Institute of Neuroinfomatics,
University / ETH Zurich,yu001@ini.phys.ethz.ch

Dept. of Electrical and Electronic Eng.,
Tokushima University
nishio@ee.tokushima-u.ac.jp

I. I NTRODUCTION
Back Propagation (BP) was introduced by Rumelhart in1986 [1]. BP is used for learning algorithm of MLP and theerror propagates backwards in the network. MLP using BP algorithm is well known to perform for the pattern classification
tasks. However, the solution of the network often falls into thelocal minimum, because BP uses the steepest descent method
for the leaning process. Generally, if the solution of MLP falls
into the local minimum, it can not escape. In order to avoid
this problem, some methods to release the solution from thelocal minimum are required.
In this study, we propose a chaos glial network whichconnects to MLP. We consider that glial cells produce chaotic
oscillation which is affected to neurons. This view is motivatedby investigations of the Hopfield network solving combinatorial optimization problems with the help of a chaotic
input signal component, designed in order to avoid local
minima. It appears, from computer simulations, that a chaotic
input component may substantially enhance the capability of
avoiding these local minima [2]-[4]. Hence, we believe thatchaotic signals may be used to further enhance the efficiencyof the proposed chaos glia neural network.
Furthermore, chaotic oscillation generated from glial cells
propagates to the neighbor glial cells. Namely, certain neuron
in this network is affected from some of glial cells located ata nearby site. We apply the proposed MLP with chaos glial
network for solving Two-Spiral Problem (TSP) and confirm
the efficiency by computer simulations.

Fig. 1.

MLP with chaos glial network.

is conventional updating rule.
n∑wij (t)xj (t) − θi (t) ,xi (t + 1) = f (1)j=1

In the chaos glial neural network, chaotic oscillation isadded to neurons in the second hidden layer. This neuron’s
updating rule is following as Eq. (2).
n∑xi (t + 1) = f wij (t)xj (t) − θi (t) + αΨi (t) ,(2)j=1

II. M ULTI -L AYER P ERCEPTRON
MLP is a most famous feed forward neural network. Thisnetwork is used for pattern recognition, pattern classification,and other tasks. MLP has some layers, it has mainly inputlayer, hidden layer, and output layer. Generally, it is known
that MLP can solve a more difficult task if the number of layeror neuron is increased. We consider MLP which is composedof four layers (one input layer, two hidden layers and oneoutput layer), and MLP has the chaos glial network in thesecond layer of the hidden layer. The proposed MLP with thechaos glial network structure (connected 2-20-40-1) is shownin Fig. 1.

where x : input or output, w : weight parameter, θ : threshold,ψ, Ψ : chaotic oscillation, α : amplitude of chaos and f :output function. And we use sigmoid for the output function
as Eq. (3).
f (a) =1,
1 + e−a(3)In the biological neural network, it is known that theglial cells affect to the neighbor neurons over a wide range
by propagating in the network [6]. In order to realize thisphenomena, we add chaotic oscillation to neurons by using
Eq. (4).

A. Neuron Updating Rule
The updating rule of neurons in the input layer, the first
hidden layer and the output layer is described by Eq. (1) which- 11 -

Ψi (t) =m∑k=m

β |k| ψi+k (t),(4)

where β denotes attenuation parameter and k is the propagating range in the glial network. Chaotic oscillation ispropagating in the glial network.
III. C HAOTIC O SCILLATION OF G LIAL N ETWORK
We use skew tent map to generate chaos. This map is oneof simple chaotic maps and the center of map is shifted for alittle from the standard tent map. This chaotic map is definedby Eq. (5). Neurons in the second hidden layer are affectedchaotic oscillation by Eq. (5) into the Eq.(4).
{ 2ψ(t)+1−A (−1 ≤ ψ(t) ≤ A)
1+A
ψi (t + 1) =,−2ψ(t)+1+A
1−A(5)(A < ψ(t) ≤ 1)

Fig. 3.

Here, we consider the case that we prepare two similar initialvalues of chaos map for adding two adjacent neurons. Figure 2shows the two time series obtained from the skew tent map
when initial values are fixed as ψ2 (0) = 0.12560 and ψ2 (0) =0.12561. From this figure, the plots of the two time series

Fig. 2.

Time Series of Skew Tent Map with Correlation.

A. Two-Spiral Problem
We apply the proposed network for solving TSP [5]. MLP
learns to each point of two spirals, and MLP learns by using
BP algorithm. Each MLP learns the two spirals by setting up

Time Series of Skew Tent Map.
Fig. 4.

are spread and difference between two time series becomeslarge with time, even if we use similar initial value to generatechaos. This is typical phenomena of chaos known as butterfly
effect. If glial cells add these chaos to neurons, each neuron
is affected with a completely different oscillation.
However, in the real biological neuro-glial network, glial
cells affect each other with neighbor cells. We use the chaos
glia propagating equation (Eq. 4) for having correlation eachother of neighbor glial cells. Figure 3 shows two chaotic timeseries by using Eq. (4). The parameters are fixed as β = 0.8,m = 5. Two chaotic time series have similar peek points. We
consider that these chaotic time series have some correlations.

IV. S IMULATIONSIn this section, the difference in the performance of ourMLP; MLP with chaos glial network and the conventional
MLP is compared.

Two-Spiral Problem.

same weights before learning process. We prepare 98 data oftwo spirals as shown in Fig. 4. The number of learning points
is fixed as 500000. We investigate the error which is modified
in the meantime. The error function is defined as Eq. (6).
1∑
E=|ti − Oi |,n i=1n(6)where E : error value t : target value, O : output value.
B. Changing the number of neuron adding chaos
We investigate the learning ability when the number ofneuron adding chaos is changed. In this simulation, we addchaos to the neurons in the second hidden layer’s. We change
the percentage of neuron adding chaos as follows 0%, 20%,50%, and 100%. The simulation result is summarized in Tab. I.
From Tab. I, we obtain the best learning ability when all neurons in the second hidden layer are added chaotic oscillation.

- 12 -TABLE I
T HE P ERFORMANCE OF MLP WITH C HAOS G LIAL N ETWORK BY
C HANGING THE N UMBER OF N EURON A DDING C HAOS
Points

0%

20%

50%

100%

1000001500002000002500003000003500004000004500005000000.306195
0.228353
0.170263
0.135390
0.120928
0.100454
0.090179
0.088718
0.094303

0.334227
0.269349
0.198626
0.153809
0.140445
0.136136
0.120175
0.121040
0.1091100.334600
0.224780
0.177127
0.147403
0.116600
0.111645
0.111659
0.094112
0.086106

0.322175
0.244479
0.166893
0.143317
0.115145
0.103821
0.081798
0.086527
0.074204

Minimum

0.018176

0.010911

0.002383

0.002523
Fig. 5.

In the next simulation, we add chaotic oscillation to every
neurons in the second hidden layer.
C. Results of Each MLP Learning
We compare MLP with chaos glial network, the conventional MLP and MLP with random glial network. In the random glial network, glial network produces random oscillation.
Each result is summarized in Tab. II.

Point

ConventionalRandomChaos

0.274507
0.189218
0.142849
0.111989
0.084422
0.093832
0.081747
0.079630
0.076502

0.299191
0.228571
0.165421
0.136988
0.118154
0.098092
0.086327
0.087916
0.083399

0.265852
0.191319
0.151007
0.118951
0.099801
0.088131
0.077735
0.071450
0.068681

Minimum

0.0125310

0.0104630

0.0072510

V. C ONCLUSION
In our study, we have proposed a chaos glial network. Thisnetwork gave chaotic oscillations to the second hidden layer’s
neuron and this chaotic oscillation propagates to other neurons.
We confirmed that MLP with chaos glial network gains better
performance than the conventional MLP and into MLP withrandom glial network for solving TSP.
R EFERENCES[1] D.E. Rumelhart, G.E. Hinton, and R.J. Williams, “Learning representations by back-propagating errors,” Nat., vol.323-9, pp.533-536, 1986.
[2] Y. Hayakawa, A. Marumoto, and Y. Sawada, “Effects of the chaotic noiseon the performance of a neural network model for optimization problems,”
Phys. Rev. E, vol. 51, no. 4, pp. 2693-2696, Apr. 1995.
[3] T. Ueta, Y. Nishio, and T. Kawabe, “Comparison between Chaotic Noiseand Burst Noise on Solving Ability of Hopfield Neural Networks,” Proc.
NOLTA’97, vol. 1, pp. 409-412, Nov. 1997.
[4] Y. Uwate, Y. Nishio, T. Ueta, T. Kawabe, and T. Ikeguchi, “Performanceof Chaos and Burst Noise Injected to the Hopfield NN for QuadraticAssignment Problems,” IEICE Transactions on Fundamentals, vol. E87A, no. 4, pp. 937-943, Apr. 2004.
[5] J.R. Alvarez-Sanchez, “Injecting knowledge into the solution of the twospiral problem,” Neural Comput. & Applic., vol.8, pp.265-272, 1999.
[6] S. Koizumi, M. Tsuda, Y. Shigemoto-Nogami, and K. Inoue, “Dynamicinhibition of excitatory synaptic transmission by astrocyte-derived ATP in
hippocampal cultures,” Proc. Natl. Acad. Sci. U. S. A, vol.100, pp.1102811028, 2003.

TABLE II
T HE P ERFORMANCE BY U SING E ACH MLP

100000150000200000250000300000350000400000450000500000The Error Curve by Three MLP Networks.

At start learning, conventional MLP is better than the others.
However, after 300000 learning points MLP with chaos glial
network gains better performance than the conventional MLP.
MLP with random glial network shows the most worst result.
If we focus on the minimum error, MLP with chaos glial
network can find smallest error value. Because similarity ofnear neurons performs well by effects of chaos glial cells. Andwhen MLP falls into local minima, chaos helps to escape witheffectively.
Figure 5 is a typical example of learning curve. MLP withrandom glial network can not learn to two spirals. MLP withchaos glial network and the conventional MLP can learn to it,
and the chaos glial network converges to the lower error value.
The learning curve of the conventional MLP is smooth. While,
the others have little vibration. We consider that this vibration
phenomena makes to escape out from the local minimum.

- 13 -