Good Learning Performance of Backpropagation Algorithmwith Chaotic Noise Features
Azian Azamimi† , Yoko Uwate‡ and Yoshifumi Nishio††Department of Electrical and Electronic Engineering
Tokushima University
2-1 Minami-Josanjima, Tokushima 770-8506, Japan‡Institute of Neuroinfomatics,
University / ETH Zurich, Switzerland
Email: {azian, nishio}@ee.tokushima-u.ac.jp, yu001@ini.phys.ethz.ch
Abstract—Over the years, many improvements
and modiﬁcations of the backpropagation learning algorithm have been reported. In this study, we propose a new modiﬁed backpropagation learning algorithm by adding the chaotic noise into weight updateprocess. By computer simulations, we conﬁrm thatthe proposed algorithm can gives a better convergence
rate and can ﬁnd a good solution in early time compared to the conventional backpropagation algorithm.
Weight update position, noise amplitude and control
parameter of chaos can give a big eﬀect on the backpropagation learning performance.

ous features of chaos can give a good eﬀect in neuralnetwork. Considering this, we try to apply some ofchaotic features to the backpropagation learning algorithm.
In this study, we propose a new modiﬁed backpropagation learning algorithm by adding the chaotic noise
into weight update process. The chaotic noise is generated by logistic map and is controlled by noise amplitude.
2. BP learning algorithm
2.1. BP batch learning algorithm

1. Introduction
In the area of artiﬁcial neural networks, the backpropagation (BP) learning algorithm has proved to beeﬃcient in many engineering applications especially inpattern recognition, signal processing and system control. The BP learning operates with a feedforward
neural network which consists of an input layer, a single or more of hidden layer and an output layer. Although the BP learning has been a signiﬁcant researcharea of neural network, it also has been known as analgorithm with a very poor convergence rate. Many
attempts have been made to the algorithm to improve
the performance on convergence speed and learningeﬃciency.
On the other hand, chaos has gained much attention
and some applications in neural network during recentyears. There have been many reports on the good
performance of Hopﬁeld neural network when chaos is
inputted to the neurons as noise [1],[2]. By computer
simulations, it has been conﬁrmed that the chaotic
noise is eﬀective for solving quadratic assigment problem and gains better performance to escape out local
minima than random noise. The authors [3] also haveproposed the feedforward neural network with chaotically oscillating gradient of the sigmoid function andproved that the proposed network can ﬁnd good solutions in early time. Hence, we consider that vari-In the standard backpropagation learning algorithm, the errors of output neurons are backpropagated through the network during training. This standard learning algorithm was introduced in [4]. The error signal of output neuron can be deﬁned by taking
the diﬀerence between the target output and the actualoutput. However, in this study, we use the batch BP
learning algorithm. The batch BP learning algorithm
is expressed by a formula similar to the standard BP
learning algorithm but the diﬀerence lies in timing of
the weight update. The weight update of the standard
BP is performed after each single input data, while forthe batch BP the weight update is performed after all
input data has been processed. The total error E ofthe network is deﬁned as follows;
P∑pN
∑
1∑
E=
Ep =(tpi − opi )2 },
{2p=1
p=1
i=1

(1)

where P is the number of the input data, N is thenumber of the neurons in the output layer, tpi denotes
the value of the desired target data for the pth input
data and opi denotes the value of the output data forthe pth input data. The goal of the learning is to set
weights between all layers of the network so that thetotal error E can be minimized. In order to minimize
the total error E, the weights are adjusted according
to the following equation:- 36 -

k−1,kk−1,kwi,j(m + 1) = wi,j(m) +P∑

k−1,k4p wi,j(m), (2)p=1
k−1,k4p wi,j(m) = −η∂Ep
k−1,k∂wi,j,(3)

k−1,kwhere wi,jis the weight between ith neuron of thelayer k − 1 and the jth neuron of the layer k, m is thelearning time and η is the learning rate. In this study,
we add the inertia term to Eq.(3) where ζ denotes theinertia rate.

k−1,k4p wi,j(m) = −η∂Ep
k−1,k∂wi,jk−1,k+ζ4p wi,j(m−1), (4)2.2. New modiﬁed BP learning algorithm
Slow convergence rate is the most common disadvantage of backpropagation learning algorithm. In
the backpropagation algorithm, the weights updates
are proportional to the error propagating from output through the sigmoid function and through theweights. The convergence rate of the learning process
can be improved by changing how the error propagatesback through the network. We believe that the weightupdate calculation is the main factor to improve thelearning process. Considering this, we add the chaotic
noise into weight update process during error propagation. The new modiﬁed weight update equation for
BP algorithm can be shown as follows where β is thenoise amplitude.
k−1,k4p wi,j(m) = −ηFigure 1: Time series obtained from logistic map(α=4.00).

∂Epk−1,k∂wi,j
k−1,k+ζ4p wi,j (m − 1) + noisei,j (m),(5)

noisei,j (m) = βi,j (m)(xi,j (m) − 0.5),(6)

The chaotic noise is generated by logistic map and
α is the control parameter of chaos. An example oftime series obtained by Eq.(7) can be shown as Fig. 1.
xi,j (m + 1) = αx0 (m)(1 − x0 (m)),(7)

3. Simulation results
In this section, we show the eﬀectiveness of chaotic
noise added into weight update process in BP algorithm by testing it to the learning example. Here, weconsider the feedforward neural network produce outputs x2 for input data x as one learning example. The
sampling range of the input data is [-1.0, 1.0]. We carried out the new modiﬁed BP learning algorithm byusing the following parameters. The learning rate andthe inertia rate are ﬁxed as η = 0.02 and ζ = 0.002 respectively. The initial values of the weights are given
between -1.0 and 1.0 at random. The learning iterations is set to 10000 and 6 neurons are prepared in
the hidden layer. The network structure and learning
example are shown in Fig. 2
It should be noted that although there maybe moresuitable values of these parameters, it is diiﬁcult to set
them theoretically. Hence, we use the same parameters during all simulations process to make it easier tocompare between the new proposed algorithm and theconventional method.
First, we investigate the learning eﬃciency byadding the chaotic noise into diﬀerent position ofweight update process. As we may know, in backpropagation learning algorithm, the main purpose ofweight adjustment is to reduce the error value between
the output and desired target. Thus, we apply thechaotic noise to diﬀerent position of weight update inorder to see the eﬀectiveness of noise adding. We add
chaotic noise to three diﬀerent positions of weight update ; a) Proposed network-1 (from input layer to hidden layer only), b) Proposed network-2 (from hidden
layer to output layer only) and c) Proposed network-3(both of them). We also compare the learning performance with the conventional method where there is nonoise application at all. Figure 3 shows the learning
performance of all proposed and conventional network
respectively. The horizontal axis is iteration time andthe vertical axis is error value.
From this ﬁgure, we can conﬁrm that the new proposed algorithm where chaotic noise is added intoweight update process gains better performance thanthe conventional algorithm when the chaos parameterof α is set to 4.0 and noise amplitude β is ﬁxed as 0.01.
The addition of chaotic noise during weight update
can help the learning process to ﬁnd a good solution
in early time compared to the conventional backpropagation algorithm. Futhermore, the diﬀerent position- 37 -(a) Network structure.
Figure 3: Learning performance for proposed and conventional network.

(b) Learning example of x2 .
Figure 2: Network structure and learning example.

of weight update also give the diﬀerent performance.
From the simulation result, we can conﬁrm that when
the chaotic noise is applied to both sides of weight update (from input layer to hidden layer and from hidden
layer to output layer), the learning process gain better
performance compared to others. Figure 4 also show
the output results for all proposed and conventional
method.
4. Conclusions
In this study, we proposed a new modiﬁed backpropagation learning algorithm by adding the chaotic noise
during weight update. By computer simulations, weconﬁrmed that addition of chaotic noise to the weight
update in learning algorithm can gives a better convergence rate and can ﬁnd a good solution of the learning
process in early time compared to the conventional
backpropagation algorithm. Diﬀerent weight update
position, the amplitude of noise and the control parameter of chaos can give a big eﬀect on the backpropagation learning performance.

Figure 4: Output results for proposed and conventional network.

Ability of Hopﬁeld Neural Networks,” Proceedingsof NOLTA’97, vol.1, pp.409-412, 1997.
[2] Y. Tada, Y. Uwate and Y. Nishio, “Performance ofChaotic Switching Noise Injected to Hopﬁeld NN
for Quadratic Assignment Problem,” Proceedingsof ISCAS’06, pp.5519-5522, 2006.
[3] Y. Uwate and Y. Nishio, “Chaotically Oscillating Sigmoid Function in Feedforward NeuralNetwork,” Proceedings of NOLTA’06, pp.215-218,
2006.
[4] D.E. Rumelhart, G.E. Hinton and R.J. Williams,“Learning Internal Representations by Error Propagation,” Parallel Distributed Processing, vol.1,
MIT Press, MA, pp.318-362, 1986.

References[1] T. Ueta, Y. Nishio and T. Kawabe, “Comparisonbetween Chaotic Noise and Burst Noise on Solving- 38 -