An Analysis of Chaotic Noise Injected to Backpropagation Algorithm
in Feedforward Neural Network
Azian Azamimi† , Yoko Uwate‡ and Yoshifumi Nishio††Department of Electrical and Electronic Engineering
Tokushima University
2-1 Minami-Josanjima, Tokushima 770-8506, Japan‡Institute of Neuroinfomatics,
University / ETH Zurich, Switzerland
Email: {azian, nishio}@ee.tokushima-u.ac.jp, yu001@ini.phys.ethz.ch
Abstract
There have been much interest in applying noise to neural networks in order to observe their eﬀect on network
performance. In our previous research, we have proposeda new modified backpropagation learning algorithm, inwhich chaotic noise is added into weight update process.
By computer simulations, we confirmed that the presenceof chaotic noise during weight update process in feedforward neural network can give a better convergence rate andcan find a good solution in early time. Hence, in this study,
we extend these results by introducing other diﬃcult learning example and analyzing the eﬀect of noise parameterssuch as noise amplitude and control parameter of chaos tothe learning performance.
1. Introduction
Backpropagation (BP) learning is one of engineering applications of artificial neural networks. The BP learning
operates with a feedforward neural network which is composed of an input layer, a single or more of hidden layers and an output layer. The eﬀectiveness of BP learning
has been recognized in many engineering applications especially in pattern recognition, system control and signalprocessing. Although the BP learning has been a significant research area of artificial neural network, it also hasbeen known as an algorithm with a poor convergence rate.
Many attempts have been made to the algorithm to improve
the performance on convergence speed and learning eﬃciency. For example, there have been a lot of reports onchanging the learning rate and also the number of neurons
in the hidden layer but this will lead to slight improvement
only [1]-[3]. Not many studies have been made on modifying the algorithm structure in order to improve the learningperformance.
On the other hand, chaos has gained much attention andsome applications in neural network over this recent years.
There have been many reports on the good performanceof Hopfield neural network when chaos is inputted to theneurons as noise [4],[5]. By computer simulations, it has

been confirmed that chaotic noise is eﬀective for solving
quadratic assigment problem and gains better performanceto escape out local minima than random noise. The authors [6] also have proposed the feedforward neural network with chaotically oscillating gradient of the sigmoid
function and proved that the proposed network can findgood solutions in early time. Hence, we consider that various features of chaos can give a good eﬀect in neural network. Considering this, we conduct a further investigationof chaotic noise eﬀect to the BP learning algorithm.
In this study, we analyze various methods of injecting
chaotic noise into BP learning algorithm. In order to confirm the generality of this work, we simulate the results using more diﬃcult functions as learning example. We also
investigate various chaotic noise parameters such as noise
amplitude and control parameter of chaos which might givediﬀerent results to the learning performance.
2. BP learning algorithm
2.1. BP batch learning algorithm
In the standard backpropagation learning algorithm, theerrors of output neurons are backpropagated through thenetwork during training. This standard learning algorithm
was introduced in [7]. The error signal of output neuron
can be defined by taking the diﬀerence between the target
output and the actual output. However, in this study, we use
the batch BP learning algorithm. The batch BP learningalgorithm is expressed by a formula similar to the standard
BP learning algorithm but the diﬀerence lies in timing of
the weight update. The weight update of the standard BP isperformed after each single input data, while for the batch
BP the weight update is performed after all input data has
been processed. The total error E of the network is defined
as follows;
E=P
∑
p=1

Ep =pN∑1∑{(t pi − o pi )2 },
2
p=1
i=1

(1)

where P is the number of the input data, N is the number ofthe neurons in the output layer, t pi denotes the value of the- 70 -

desired target data for the pth input data and o pi denotes
the value of the output data for the pth input data. Thegoal of the learning is to set weights between all layers of
the network so that the total error E can be minimized. Inorder to minimize the total error E, the weights are adjusted
according to the following equation:

11

0.8x(m+1)
0.6

0.5x(m)0.4

0.2000k−1,kwk−1,ki, j (m + 1) = wi, j (m) +P∑

0.5x(m)(2)∂E p∂wk−1,ki, j,∂E p∂wk−1,ki, j+ ζ4 p wk−1,ki, j (m − 1),

400

600

800

1000(b) Time series.

Figure 1: Skew tent map.

(3)

where wk−1,kis the weight between ith neuron of the layeri, jk − 1 and the jth neuron of the layer k, m is the learning
time and η is the learning rate. In this study, we add theinertia term to Eq.(3) where ζ denotes the inertia rate.
4 p wk−1,ki, j (m) = −η200time(a) Skew tent map.
4 p wk−1,ki, j (m),p=1

4 p wk−1,ki, j (m) = −η01(4)2.2. Chaotic BP learning algorithm

we consider the feedforward neural network produce outputs A sin(ωt − ϕ) + D for input data x as one learning example. The sampling range of the input data is [-1.0, 1.0].
We carried out the chaotic BP learning algorithm by using
the following parameters. The learning rate and the inertia
rate are fixed as η = 0.1 and ζ = 0.001 respectively. The initial values of the weights are given between -1.0 and 1.0 at
random. The learning iterations is set to 20000 and 8 neurons are prepared in the hidden layer (Fig. 2). The learning
example of sinwave functions with diﬀerent amplitude areshown in Fig. 3.

In our previous research, we have proposed a new modified BP learning algorithm, namely chaotic BP learningalgorithm. This new algorithm can be expressed by a similar formula with the standard BP algorithm but the diﬀerent lies in weight update process. Chaotic noise is added
into weight update process during error propagation. The
weight update process for chaotic BP learning algorithm
can be shown as follows, which β denotes the noise amplitude.
4 p wk−1,ki, j (m) = −η∂E p∂wk−1,ki, j+ζ4 p wk−1,k(m−1) + noisei, j (m),i, j(5)noisei, j (m) = βi, j (m)(xi, j (m) − 0.5),(6)

Figure 2: Network structure.

Chaos, xi, j (m) is generated by skew tent map and α isthe control parameter of chaos. The skew tent map andan example of time series obtained by Eq.(7) are shown inFig. 1.
 xi, j (m)


 α
xi, j (m + 1) =  xi, j (m)−1α−1sinwave
1
y=0.5sin(2πx-π/6)+0.5
0.8
y=0.4sin(2πx-π/6)+0.40.60.4(0 ≤ xi, j (m) ≤ α)y=0.25sin(2πx-π/6)+0.25

0.2(7)(α < xi, j (m) ≤ 1)

0-1-0.5

00.5

1xFigure 3: Learning example of sinwave with diﬀerent amplitude.

3. Simulation results
In order to confirm the generality of this chaotic BP
learning algorithm, we simulate it to various sinwave functions with diﬀerent amplitude as learning example. Here,- 71 -It should be noted that although there maybe more suit-able values of these parameters, it is diificult to set them
theoretically. Hence, we use the same parameters during
all process to make it easier to analyze the simulation results.

Figure 5 show the learning performance of all proposedand conventional network for diﬀerent sinwave functions..
The horizontal axis is iteration time and the vertical axis iserror value.

3.1. Random noise vs. chaotic noise
In order to see the eﬀectiveness of chaotic noise to thelearning performance, we make a comparison between random noise and chaotic noise. We observe their performance by simulating it to various sinwave functions with
diﬀerent amplitude. The noise amplitude, β is fixed as
0.01. The learning performance for both random noise andchaotic noise are shown in Fig. 4.
The results show that the chaotic noise has much better
performance than random noise. The random noise cannot escape local minima while the chaotic noise can give afaster convergence rate.

(a) Sinwave 1.

error
random noisechaotic noise

0.06(b) Sinwave 2.

(c) Sinwave 3.

0.04

Figure 5: Learning performance of proposed and conventional network for diﬀerent sinwave example .

0.0200500010000
iterations

15000

20000

From this figure, we confirm that our proposed network
which chaotic noise is added into the weight update gains
random noise
0.04
random noise
better performance than the conventional method when thechaotic noisechaotic noise
0.03
chaos parameter of α is set to 0.55 and noise amplitude
0.03
β is fixed as 0.01. The addition of chaotic noise during
0.02
0.02
weight update can help the learning process to find a good
0.01
0.01
solution in early time compared to the conventional backpropagation algorithm. Futhermore, the diﬀerent position
0005000
1000015000
20000
05000
10000
15000
20000
iterations
iterationsof weight update also give the diﬀerent performance. Fromthe simulation results, we confirm that when the chaotic(b) Sinwave 2.
(c) Sinwave 3.
noise is applied to both sides of weight update (from inFigure 4: Comparison of learning result between random put layer to hidden layer and from hidden layer to output
noise and chaotic noiselayer), the learning process gain better performance compared to others. Figure 6 also show the output results for
all proposed and conventional method.
(a) Sinwave 1.

errorerror
0.04

3.2. Noise added to diﬀerent weight update positions
3.3. Noise amplitude and control parameter of chaos
We also investigate the learning eﬃciency by adding
chaotic noise into diﬀerent position of weight update. As
we may know, in backpropagation learning algorithm, themain purpose of weight adjustment is to reduce the error
value betweet output and desired target. Thus, we apply thechaotic noise to diﬀerent position of weight update in order
to see the eﬀectiveness of noise adding. We add chaotic
noise to three diﬀerent positions of weight update ; a) Proposed network-1 (from input layer to hidden layer only),b) Proposed network-2 (from hidden layer to output layeronly) and c) Proposed network-3 (both of them). We also
compare the learning performance with the conventional
method which there is no chaotic noise application at all.

We also study the eﬀect of noise amplitude and chaosparameter to the learning performance. The diﬀerent valueof noise amplitude give the diﬀerent learning results. Here,
we only choose one sinwave example to compare the diﬀerent value of noise amplitude. Figure 7 show the learning
performance when the noise amplitude β is set to 0.001,0.01 and 0.1 respectively.
Choosing the approriate noise amplitude value is important for this work because it will eﬀect the learning performance. From this results, we can see that when β is fixed
as 0.001, there will be no improvement in learning performance because the noise amplitude is too small. On the- 72 -(a) β=0.001.

(a) Sinwave 1.

(b) Sinwave 2.

(b) β=0.01.

(c) Sinwave 3.

Figure 6: Output results of proposed and conventional network for diﬀerent sinwave example.

other hand, when β value is set to 0.1, the learning performance become not stable due to the large value of noise
amplitude. Hence, we understand that the proposed network give good results if noise amplitude is set to 0.01.
Furthermore, we also investigate the eﬀect of chaos parameter α to the learning performance. Figure 8 show
the learning performance when chaos parameter α is setto 0.50 and 0.55 respectively. From skew tent map bifurcation, we know that chaos parameter α=0.55 show the fullydeveloped chaos. We confirm that the fully-developed
chaos gains better learning performance.
4. Conclusions
In this study, we have conducted a further analysis ofchaotic noise to the feedforward neural network. We havesimulated the proposed method by introducing other difficult learning example and confirmed that the addition of
chaotic noise into weight update process in backpropagation learning algorithm can gives a better convergence ratecompared to standard backpropagation algorithm. We also
found that diﬀerent weight update position, noise amplitude and control parameter of chaos can give a big eﬀect
on the backpropagation learning performance.
References[1] A. Sperduti and A. Starita, “Speed up learning and network optimization with extended back-propagation,”
Neural Networks, vol.6, pp.365-383, 1993.
[2] A. Miniani and R.D. Williams, “Acceleration of backpropagation through learning rate and momentum
adaption,” Proceedings of IJCNN’90,San Diego,CA,
pp.676-679, 1990.

(c) β=0.1.

Figure 7: Learning performance of proposed and conventional network with diﬀerent value of noise amplitude .

(a) α=0.50.

(b) α=0.55.

Figure 8: Learning performance of proposed and conventional network with diﬀerent value of chaos parameter.

[3] R.A. Jacobs, “Increased rates of convergence thorugh learning rate adaption,” Neural Networks, vol.1,
pp.295-308, 1988.
[4] T. Ueta, Y. Nishio and T. Kawabe, “Comparison between Chaotic Noise and Burst Noise on Solving
Ability of Hopfield Neural Networks,” Proceedings ofNOLTA’97, vol.1, pp.409-412, 1997.
[5] Y. Tada, Y. Uwate and Y. Nishio, “Performance ofChaotic Switching Noise Injected to Hopfield NN for
Quadratic Assignment Problem,” Proceedings of ISCAS’06, pp.5519-5522, 2006.
[6] Y. Uwate and Y. Nishio, “Chaotically Oscillating Sigmoid Function in Feedforward Neural Network,” Proceedings of NOLTA’06, pp.215-218, 2006.
[7] D.E. Rumelhart, G.E. Hinton and R.J. Williams,“Learning Internal Representations by Error Propagation,” Parallel Distributed Processing, vol.1, MITPress, MA, pp.318-362, 1986.

- 73 -