Back Propagation Learning of Neural Networks
with Chaotically-Selected Affordable Neurons
Yoko Uwate and Yoshifumi Nishio
Dept. of Electrical and Electronics Engineering, Tokushima University
2-1 Minami-Josanjima, Tokushima, Japan
Email: {uwate, nishio}@ee.tokushima-u.ac.jp

Abstract— Cell assembly is one of explanations of information
processing in the brain, in which an information is represented
by a firing space pattern of a group of plural neurons. On theother hand, effectiveness of neural network has been confirmed inpattern recognition, system control, signal processing, and so on,
since the back propagation learning was proposed. In this study,
we propose a new network structure with chaotically-selectedaffordable neurons in the hidden layer of the feedforward neuralnetwork. Computer simulated results show that the proposed
network exhibits a good performance for the back propagation
learning.

I. I NTRODUCTION
Recently, studies on the brain have been carried out actively
on various levels. On the system level, many researchers investigate how the brain operates to realize higher functions suchas learning, memory, emotion, and so on. Cell assembly [1]-[3]is one of explanations of information processing in the brain, in
which an information is represented by a firing space pattern ofa group of plural neurons. This mechanism has been proposed
by Hebb who is a psychologist in 1949. Cell assembly could
achieve multiple information processing carried out at thesame time and parallel dispersion processing of the brain byforming a functional circuit with some neurons according to
the necessary processing. Although this concept is impressive,
we consider that more complex mechanism is realized in thebrain.
On the other hand, Back Propagation (BP) learning [4] isone of engineering applications of neural networks. The BP
learning operates with a feedforward neural network which iscomposed of an input layer, a hidden layer and an output layer,
and the effectiveness of the BP learning has been confirmed inpattern recognition, system control, signal processing, and soon [5]-[7]. When the BP learning is applied, the hidden layer
plays the essential role to decide the performance of the neuralnetwork. Hence, for example, there have been a lot of reportson the number of the neurons in the hidden layer. However,there have not been many studies on changing the structure of
the hidden layer to improve the performance on convergence
speed or learning efficiency.
In this study, we propose a new network model which
is a feedforward neural network with chaotically-selectedaffordable neurons in the hidden layer for more efficient BP
learning. In this network, we prepare some extra neurons in
the hidden layer. When the network executes the BP learning,
all of the neurons in the hidden layer are not used at every

0-7803-8834-8/05/$20.00 ©2005 IEEE.

updating. Namely, some of the neurons are selected for thelearning and the rest of the neurons are deactivated. Further,
the selected set of the neurons is changed in a chaotic mannerat every updating. Computer simulated results show that theproposed network exhibits a good performance for the BP
learning on both convergence speed and learning efficiency.
II. N ETWORK M ODEL WITH C HAOTICALLY-S ELECTEDA FFORDABLE N EURONS
Cell assembly is one of explanations of information processing in the brain, in which an information is represented
by a firing space pattern of a group of plural neurons. Thismechanism could achieve multiple information processing
carried out at the same time and parallel dispersion processingof the brain by forming a functional circuit with some neurons
according to the necessary processing. Conceptual diagramof cell assembly is shown in Fig. 1. Informations A and
B are expressed by the corresponding functional connections
between plural neurons. It is a special characteristics of cell
assembly to exist overlapping neurons for the informations
A and B. Although this concept is impressive, we consider
that more complex mechanism is realized in the brain. For
example, in the case of cell assembly, an information A isexpressed by a specific group of the neurons. However, weconsider that the same information A can be expressed byseveral different groups of neurons in the brain.

AFig. 1.

B

Conceptual diagram of cell assembly.

A. Network Model with Affordable Neurons
We introduce affordable neurons in the hidden layer of thefeedforward neural network to reflect a function of the brain.
The extra neurons in the hidden layer are prepared in advance.
During the BP learning, all of the neurons in the hidden layer
are not used at every updating. Namely, some of the neurons1481

1

are selected for the learning and the rest of the neurons aredeactivated. The network model with the affordable neurons
is shown in Fig. 2.

x n+10

Chaos

Chaos

Chaos

Chaos

-1-1

Chaos

Chaos

0 a=0.05

Output layer

Input layer

Fig. 3.

xn1

Skew tent map.

Chaos

Chaos

Affordable
neuronHidden layer

Fig. 2.

Network model with affordable neurons.

B. Chaotic Selection
For the proposed network, some of the neurons have to be
selected at every updating of the BP learning. The authors
have investigated the performance of the Hopfield neuralnetwork solving combinatorial optimization problems whenchaos is inputted to the neurons as noise [8]-[10]. By computer
simulations, chaotic noise has been confirmed to gain better
performance to escape out of local minima than random noise.
Hence, we consider that various features of chaos are effective
for neural networks.
In this study, we use the skew tent map as a simple chaotic
map to realize chaotic selection of operating neurons in thehidden layer. We prepare the skew tent maps with different
initial values, whose number is the same as the number ofthe neurons in the hidden layer, and each skew tent map iscorresponded to each neuron. At the every updating of the BP
learning, the skew tent maps are also updated and their values
are referred. We select a certain number of the neurons in theorder of the values of the skew tent maps. In this case, theselection is made in a chaotic manner. Note that chaos is not
inputted to the neural network directly, but is used only forthe selection of the operating neurons. The skew tent map isdefined by the following equation and the map is shown in
Fig. 3.
2xn + 1 − a(−1 ≤ xn ≤ a)
1+a(1)
xn+1 =
 −2xn + 1 + a (a < xn ≤ 1)
1−aIII. BP L EARNING A LGORITHM
The standard BP learning algorithm was introduced in[4]. The BP is the most common learning algorithm for
feedforward neural networks. In this study, we use the batch
BP learning algorithm. The batch BP learning algorithm isexpressed by similar formula of the standard BP learningalgorithm. The difference lies in the timing of the update ofthe weight. The update of the standard BP is performed after
each single input data, while the update of the batch BP is
performed after all different input data.
The total error E of the network is defined as the followingequation.
 NPP1
Ep =(tpi − opi )2 )
(2)
E=2
p=1
p=1
i=1
where P is the number of the input data, N is the number ofthe neurons in the output layer, tpi denotes the value of thedesired target data for the pth input data, and opi denotes thevalue of the output data for the pth input data. The goal of thelearning is to set weights between all layers of the network
to minimize the total error E. In order to minimize E, theweights are adjusted according to the following equation:
k−1,kk−1,k(m + 1) = wi,j(m) +wi,jk−1,k∆p wi,j(m) = −η

∂EpPp=1

k−1,k∆p wi,j(m)

(3)

k−1,k
∂wi,jk−1,kwhere wi,jis the weight between the ith neuron of the layerk −1 and the jth neuron of the layer k, m is the learning time,and η is a proportionality factor known as the learning rate.
In this study, we introduce inertia term in the 2nd term of theright-hand side of Eq. (4).
k−1,k∆p wi,j(m) = −η∂Epk−1,k∂wi,j

k−1,k+ ζ∆p wi,j(m − 1)(4)

where ζ denotes the inertia rate.
IV. S IMULATED R ESULTS
We consider the feedforward neural network producing
outputs x2 for inputs data x as one learning example. Thesampling range of the input data is [−1.0, 1.0] and the step
size of the input data is set to be 0.01. We carried out the BP
learning by using the following parameters. The parameterof the inertia rate is fixed as η = 0.03 and initial values of
the weights are given between −1.0 and 1.0 at random. The
learning time is set to 20000. We investigate the convergence1482

speed and the learning efficiency as the total error between
the output and the desired target, when the network structureof the hidden layer is changed as follows.
[Hidden:2] 2 neurons in the hidden layer.
[Hidden:6] 6 neurons in the hidden layer.
[Hidden:8] 8 neurons in the hidden layer.
[Hidden:8(6)] 8 neurons are prepared in the hidden layer andonly 6 neurons are operated at every update time.
The simulation results are shown in Fig. 4. The horizontal
axis is the learning time and the vertical axis is the totalerror. We can see both of the convergence speed and thelearning efficiency from the figure. Figures 4(a), (b) and (c)show the cases of η = 0.01, 0.1 and 0.6, respectively. In
each figure, four graphs representing the results of the fourcases [Hidden:2], [Hidden:6], [Hidden:8] and [Hidden:8(6)]are shown.
When the learning rate is η = 0.01 (Fig. 4(a)), we confirmedthat the all of the learning curves are similar. In this case, thenetwork goes down to a local minimum corresponding to astable state very slowly. Therefore, we can confirm that theconvergence speed is slow and the learning efficiency is not
good. (Note that the scales of the vertical axes are different in
the figures.)
When the learning rate is η = 0.1 (Fig. 4(b)), we confirmedthat the improvement of the learning efficiency is difficult
by only increasing the number of the neurons in the hidden layer. However, the proposed network with chaoticallyselected affordable neurons in the hidden layer [Hidden:8(6)]gains a good performance. We consider that the network withchaotically-selected affordable neurons has an advantage toescape from local minima corresponding to some stable states
because of its chaotic behavior.
When the learning rate is η = 0.6 (Fig. 4(c)), we confirmedthat the learning curves of the traditional network becomes
oscillatory and can not converge. However, the learning curveof the proposed network with chaotically-selected affordable
neurons [Hidden:8(6)] is not oscillatory and converges quickly.
From these results, we can say that the proposed networkwith chaotically-selected affordable neurons gains a good
performance for the BP learning on both convergence speed
and learning efficiency.
V. OTHER SELECTION METHODSIn order to confirm the effectiveness of the chaotic selectionof the affordable neurons, in this section, we investigate theperformance of the BP learning of the network with affordable
neurons selected other non-chaotic methods, namely regularmethod and random method. For the regular method, theoperating neurons in the hidden layer are selected in order.
For the random method, the operating neurons in the hidden
layer are selected completely at random.
The simulation results are shown in Fig. 5. Figures 5(a), (b)and (c) show the cases of η = 0.01, 0.1 and 0.6, respectively.
In each figure, three graphs representing the results of the threecases “chaos,” “regular” and “random” methods are shown.

From the figures, we can confirm that the learning curve forthe regular method becomes oscillatory and can not converge
for any η. Namely, if the affordable neurons are selected in a
regular way, the BP learning does not succeed. We consider
that this is very interesting result stimulating our motivation
to investigate the proposed network more deeply from theviewpoint of learning process of real brain.
The learning curves for both of the chaos and the random
methods are not oscillatory and converge to certain error
values. For small learning rate in Fig. 5(a), the two curves
are similar and they converge to similar value. This is because
the network goes down to a stable state very slowly. For larger
learning rates in Figs. 5(b) and (c), the learning curve for thechaos method shows much better performance than the random
method. Namely, the random method does not oscillate, butdoes not gain a good learning efficiency. We consider that thechaos method is better than the random method in the sensethat chaotic behavior has an advantage over random behavior
to escape from local minima corresponding to some stablestates.
VI. C ONCLUSION
In this study, we have proposed a new network model which
was a feedforward neural network with chaotically-selectedaffordable neurons in the hidden layer for more efficient BPlearning. Computer simulated results showed that the proposed
network with chaotically-selected affordable neurons gained agood performance for the BP learning on both convergence
speed and learning efficiency. Further, we confirmed that thechaotic selection was important to obtain the good performance.
R EFERENCES[1] Y. Sakurai, “Dependence of Functional Synaptic Connections of Hippocampa and Neocortical Neurons on Types of Memory,” Neuroscience
Letters, vol. 158, pp. 181-184, 1993.
[2] E. Vaadia, I. Haalman, M. Abeles, H. Bergman, Y. Prut, H. Slovin and A.
Aertsen, “Dynamics of Neural Interactions in Monkey Cortex in Relation
to Behavioral Events,” Nature, vol. 373, pp. 515-518, Feb. 1995.
[3] Y. Sakurai, “Hippocampa and Neocortical Cell Assemblies Encode Memory Processes for Different Types of Stimuli in the Rat,” Journal ofNeuroscience, vol. 16, pp. 2809-2819, 1996.
[4] D.E. Rumelhart, G.E. Hinton and R.J. Williams, “Learning Internal
Representations by Error Propagation,” Parallel Distributed Processing,vol. 1, MIT Press, MA, pp. 318-362, 1986.
[5] Y. LeCun, B. Boser, J.S. Denker, et al, “Handwritten Digit Recognition
with a Back-Propagation Network,” Advances in Neural Information
Processing Systems, vol. 2, MIT Press, MA, 1990.
[6] M. Kawano, Y. Uno, M. Isobe and R. Suzuki, “A Hierarchical Neural
Network Model for Voluntary Movement with Application to Robotics,”
IEEE Control Systems Magazine, vol. 8, pp. 8-16, 1988.
[7] T.J. Sejnowski and C.R. Rosenberg, “Parallel Networks that Learn toPronounce English Text,” Complex Systems, vol. 1, pp. 145-168, 1987.
[8] T. Ueta, Y. Nishio and T. Kawabe, “Comparison between Chaotic Noiseand Burst Noise on Solving Ability of Hopfield Neural Networks,” Proc.
NOLTA’97, vol. 1, pp. 409-412, Nov. 1997.
[9] Y. Uwate, Y. Nishio and A. Ushida, ”Markov Chain Modeling of Intermittency Chaos and its Application to Hopfield NN,” IEICE Transactions
on Fundamentals, vol. E87-A, no. 4, pp. 774-779, Apr. 2004.
[10] Y. Uwate, Y. Nishio, T. Ueta, T. Kawabe and T. Ikeguchi, “Performanceof Chaos and Burst Noise Injected to the Hopfield NN for Quadratic
Assignment Problems,” IEICE Transactions on Fundamentals, vol. E87A, no. 4, pp. 937-943, Apr. 2004.

1483

0.1

Hidden:2
Hidden:6
Hidden:8
Hidden:8(6)

Error

Error

0.01

0.001

0.0001

Hidden:8(6)-Chaos
Hidden:8(6)-Regular
Hidden:8(6)-Random

0.1

0.01

0.001

0.0001
05000

10000
t

15000

20000

05000

10000
t

15000

20000

(a) Learning rate η = 0.01.

(a) Learning rate η = 0.01.
10.1
Hidden:2
Hidden:6Hidden:8
Hidden:8(6)0.01

Hidden:8(6)-Chaos
Hidden:8(6)-Regular
Hidden:8(6)-Random

0.1

Error

Error

0.01
0.001

0.001

0.00010.00010.00001
0.00001
05000

10000
t

15000

020000

5000

10000
t

15000

20000

(b) Learning rate η = 0.1.

(b) Learning rate η = 0.1.
11
Hidden:2
Hidden:6
Hidden:8
Hidden:8(6)Hidden:8(6)-Chaos
Hidden:8(6)-Regular
Hidden:8(6)-Random
0.1

Error

Error

0.1

0.01

0.01

0.001

0.001
05000

10000
t

15000

20000

Convergence of learning for different network structures.

5000

10000
t

15000

20000

(c) Learning rate η = 0.6.

(c) Learning rate η = 0.6.
Fig. 4.

0Fig. 5. Convergence of learning for different selection methods of affordable
neurons.

1484