2009 International Workshop on Nonlinear Circuits and Signal Processing
NCSP'09, Waikiki, Hawaii, March 1-3, 2009

An Improvement in Pattern Recognition Problem
using Chaotic BP Learning Algorithm
Azian Azamimi† , Yoko Uwate‡ and Yoshifumi Nishio††Tokushima University
2-1 Minami-Josanjima, Tokushima, Japan
Phone/FAX:+81-88-656-7501
Email: azian, nishio@ee.tokushima-u.ac.jp‡Institute of Neuroinfomatics,
University / ETH Zurich, Switzerland
Phone/FAX:+41 (0)44 6353053
Email: yu001@ini.phys.ethz.ch

Abstractrons as noise [6]-[8]. By computer simulations, it has been
confirmed that chaotic noise is eﬀective for solving quadratic
Pattern recognition is one of the most diﬃcult problems and assigment problem and gains better performance to escapesomehow impossible to be solved by conventional comput- out local minima than random noise. Hence, we consider thaters. However, it can be solved by using a feed forward neural various features of chaos can give a good eﬀect in neural netnetwork that has been trained accordingly. In our previous work.
research, we have proposed a new modified BP algorithm,
In our previous research, we have proposed a new modnamely Chaotic BP learning algorithm [1][2]. Simulated re- ified BP learning algorithm (Chaotic BP), in which chaoticsults show that our proposed algorithm obtain a good solution noise is added into weight update process during error propain early time when simple learning examples are used. Hence, gation. Computer simulation results show that the addition ofin this study, we implement our proposed algorithm to learn a chaotic noise during weight update process can give a faster
pattern recognition task which is an example of diﬃcult learn- learning performance and a better convergence rate compareding examples. By computer simulations, we confirmed that with standard BP algorithm. Hence, in this study, we extend
our proposed algorithm can give a better recognition rate and our investigation by using our proposed algorithm to learn
less iteration time compared with standard BP algorithm.
a pattern recognition problem. We also investigate on various factors such as the role of hidden neurons and certain
parameters which might give diﬀerent results to the learning
1. Introductionperformance.
Back Propagation (BP) learning is one of engineering applications of artificial neural networks. The BP learning operates with a feedforward neural network which is composed of
an input layer, a single or more of hidden layers and an output
layer. The eﬀectiveness of BP learning has been recognizedin many engineering applications especially in pattern recognition, system control and signal processing. Although theBP learning has been a significant research area of artificial
neural network, it also has been known as an algorithm with apoor convergence rate. Many attempts have been made to thealgorithm to improve the performance on convergence speed
and learning eﬃciency. For example, there have been a lotof reports on changing the learning rate and also the numberof neurons in the hidden layer but this will lead to slight improvement only [3]-[5]. Not many studies have been madeon modifying the algorithm structure in order to improve thelearning performance.
On the other hand, chaos has gained much attention andsome applications in neural network over this recent years.
There have been many reports on the good performance of
Hopfield neural network when chaos is inputted to the neu-

2. Chaotic BP Learning Algorithm
The chaotic BP algorithm can be expressed by a similar formula with the standard BP algorithm but the diﬀerent
lies in weight update process. In the standard BP learningalgorithm, the errors of output neurons are backpropagatedthrough the network during training. This standard learningalgorithm was introduced in [9]. The error signal of output
neuron can be defined by taking the diﬀerence between thetarget output and the actual output as shown in equation below:
pP
N∑∑
1∑
E=Ep ={(1)(t pi − o pi )2 },
2
p=1
p=1
i=1
where P is the number of the input data, N is the number ofthe neurons in the output layer, t pi denotes the value of thedesired target data for the pth input data and o pi denotes thevalue of the output data for the pth input data. The goal of thelearning is to set weights between all layers of the network sothat the total error E can be minimized. In order to minimize- 213 -

the total error E, the weights are adjusted according to thefollowing equation:
k−1,kwk−1,ki, j (m + 1) = wi, j (m) +P∑

4 p wk−1,ki, j (m),(2)p=1

4 p wk−1,ki, j (m) = −η∂E p∂wk−1,ki, j,(3)where wk−1,kis the weight between ith neuron of the layeri, j
k − 1 and the jth neuron of the layer k, m is the learning timeand η is the learning rate. In this study, we add the inertia
term to Eq.(3) where ζ denotes the inertia rate.
(a) Time series of Bernoulli shift map.

4 p wk−1,ki, j (m) = −η∂E p∂wk−1,ki, j+ ζ4 p wk−1,ki, j (m − 1),(4)However, in our proposed algorithm, we add chaotic noise
into weight update process during error propagation. The
weight update process for chaotic BP learning algorithm canbe shown as follows, which β denotes the noise amplitude.
4 p wk−1,ki, j (m) = −η∂E p∂wi,k−1,kj
k−1,k+ζ4 p wi, j (m − 1) + noisei, j (m),(5)

noisei, j (m) = βi, j (m)(xi, j (m) − 0.5),(6)(b) Bernoulli shift map.

2.1. Chaotic noise
Chaos can be generated by using various chaotic maps. In
our previous research, we have introduced logistic map andskew tent map, which are well-known as one-dimensional
simple chaotic map to generate chaos. In order to confirm theeﬀectiveness of our proposed algorithm, we introduce otherchaotic map as well. Hence, in this study, we use anotherone-dimensional chaotic map which is Bernoulli shift map togenerate chaos.
An example of time series and Bernoulli shift map obtainedby equation below are shown in Fig. 1. One of the most important parameters for this work is α, the control parameterof chaos.
 xi, j (m)(0 ≤ xi, j (m) ≤ α)
 αxi, j (m + 1) =  −xi, j (m)+α (α ≤ xi, j (m) ≤ 1)
α−1(7)

Figure 1: Bernoulli shift map.

3. Simulation results
Here, we consider a pattern recognition task, as one
example of diﬃcult problems. The learning example ofpattern recognition is 10 patterns of numeric characters(1, 2, 3, 4, 5, 6, 7, 8, 9, 0) and each pattern is composed of 7x5
neurons (see Fig. 2.) This numeric characters are fed into theneural network for recognition. In this case, the number ofneurons in the input layer is 35 and the number of neurons
in output layer is fixed to 10. We choose 16 neurons in thehidden layer. For recognition, a set of 10 patterns shifted 1
bit from each original pattern was prepared, leading a set of
100 patterns to be recognized.
We carried out the chaotic BP learning algorithm by using
the following parameters. The learning rate and the inertia- 214 -rate are fixed as η = 0.1 and ζ = 0.001 respectively. The
initial values of the weights are given between -1.0 and 1.0 at
random. The learning iterations is set to 20000 steps. Here,chaos parameter of α is set to 0.55 and noise amplitude β is
fixed as 0.01.

Figure 3: Digram model of all proposed network.
Figure 2: Pattern recognition.

3.1. Noise added into diﬀerent weight update positionsof chaotic noise during weight update process can help thelearning process to find a good solution in early time compared with the conventional BP algorithm although the improvement rate is very small.

Error rate

First, we investigate the learning eﬃcacy by adding chaotic
noise into diﬀerent position of weight update. As we may
1.2
conventional network
know, in BP learning algorithm, the main purpose of weight
proposed network-1
proposed network-2
update is to reduce the error value between output and desired
1
proposed network-3
target. We add chaotic noise into three diﬀerent positionsof weight update ; a) Proposed network-1 (from input layer
0.8
to hidden layer only), b) Proposed network-2 (from hidden
0.6
layer to output layer only) and c) Proposed network-3 (bothof them). The digram model of our proposed network can be
0.4
shown in Fig. 3.
We make a comparison between our proposed network and0.2
the conventional BP algorithm where there is no noise adding
at all. Figure 4 shows the learning performance of all pro005000
10000
15000
20000
posed and conventional BP network for pattern recognition
iteration
time
problem in term of error rate calculation. We conduct 10 trials by using diﬀerent initial conditions and we obtain reliableaverages of error rate. The final value of error rate at itera- Figure 4: Learning curve of all conventional and proposed
networks for pattern recognition.
tions=20000 can be summarized as below.
Table 1: Final value of error rate.
Conventional Network-1 Network-2 Network-3
0.5362454
0.2259438 0.3859357 0.1982767

3.2. Role of hidden neurons

From this result, we confirm that our proposed network
which chaotic noise is added into the weight update process can give better learning performance than the conventional network for pattern recognition problem. The addition

During network training, the number of hidden neurons are
crtitical. It is diﬃcult to set an accurate number of hidden
neurons. Therefore, in this study we conduct a further investigation on how to optimize the hidden layer neurons.
Figure 5 displays the learning curve of proposed network3 when the number of hidden neurons are changed. From this- 215 -figure, we can relate the network learning ability with hidden layer neurons. Upon increasing the number of hidden
neurons, the learning ability also increase, showing that thehidden neurons play an important role during learning process. But if the number of hidden neurons are too large, itwill lead the learning performance to oscillate. Here, the optimum number of hidden neurons are 16.

[4] A. Miniani and R.D. Williams, “Acceleration of backpropagation through learning rate and momentum adaption,” Proceedings of IJCNN’90,San Diego,CA, pp.676679, 1990.
[5] R.A. Jacobs, “Increased rates of convergence throughlearning rate adaption,” Neural Networks, vol.1, pp.295308, 1988.

1.2

Error rate

[6] T. Ueta, Y. Nishio and T. Kawabe, “Comparison between
Chaotic Noise and Burst Noise on Solving Ability ofHopfield Neural Networks,” Proceedings of NOLTA’97,
vol.1, pp.409-412, 1997.

hidden neurons = 4
hidden neurons = 8
hidden neurons = 12
hidden neurons = 16
hidden neurons = 20

10.8

[7] Y. Tada, Y. Uwate and Y. Nishio, “Performance of
Chaotic Switching Noise Injected to Hopfield NN for
Quadratic Assignment Problem,” Proceedings of ISCAS’06, pp.5519-5522, 2006.

0.6

0.4

0.2

005000

10000

15000

20000

iteration time

Figure 5: Learning curve of proposed network-3 when thenumber of hidden neurons are changed.

[8] Y. Uwate and Y. Nishio, “Chaotically Oscillating Sigmoid Function in Feedforward Neural Network,” Proceedings of NOLTA’06, pp.215-218, 2006.
[9] D.E. Rumelhart, G.E. Hinton and R.J. Williams, “Learning Internal Representations by Error Propagation,” Parallel Distributed Processing, vol.1, MIT Press, MA,
pp.318-362, 1986.

4. Conclusions
In this study, we have applied our proposed algorithm
to learn a pattern recognition which is an example of diﬃcult problems. From simulation results, we confirmed thatour proposed network can improve the learning performancecompared with the conventional BP algorithm. We confirmedthat the addition of chaos noise into weight update processduring error propagation can give a better recognition rateand less iteration time. We also discovered that the learning process will take a longer time if more diﬃcult learning
examples are used compared with the easy problems.
References[1] A. Azamimi, Y. Uwate and Y. Nishio, “Good Learning Performance of Backpropagation Algorithm withChaotic Noise Features,” Proceedings of SSJW”08,pp.
36-38, Aug. 2008.
[2] A. Azamimi, Y. Uwate and Y. Nishio, “An Analysisof Chaotic Noise Injected to Backpropagation Algorithm in Feedforward Neural Network,” Proceedings ofIWVCC’08, pp. 70-73, Nov 2008.
[3] A. Sperduti and A. Starita, “Speed up learning andnetwork optimization with extended back-propagation,”
Neural Networks, vol.6, pp.365-383, 1993.

- 216 -