2008 IEEE Workshop on
Nonlinear Circuit Networks, Tokushima, Dec 5-6, 2008

Chaotic BP Algorithm for Sine Wave Function
Learning with Different Frequencies
Azian Azamimi and Yoshifumi Nishio

Yoko Uwate

Dept. of Electrical and Electronic Eng.,
Tokushima University,Tokushima, 770-8506 JAPAN
Email:azian, nishio@ee.tokushima-u.ac.jp

Institute of Neuroinfomatics,
University / ETH Zurich, Switzerland
Email: yu001@ini.phys.ethz.ch

Abstract— Stochastic resonance is observed when noise added
to a system improves the systems performance in some ways.
Using this concept, we apply noise to neural networks in orderto observe their effect on network performance. In our previous
research, we have proposed a new modified back propagation(BP) learning algorithm, in which chaotic noise is added intoweight update process [1][2]. Computer simulation results showthat the addition of chaotic noise during weight update process
can give a faster learning performance and a better convergence
rate compared to the standard BP algorithm. Hence, in thisstudy, we extend our investigation by simulating our proposed
network to learn sine wave function with different frequencies
as one learning problem. We observe and analyze the network
performance in order to confirm the generality of this work.

I. I NTRODUCTION
Backpropagation (BP) learning is one of engineering applications of artificial neural networks. The BP learning operates
with a feedforward neural network which is composed of aninput layer, a single or more of hidden layers and an outputlayer. The effectiveness of BP learning has been recognizedin many engineering applications especially in pattern recognition, system control and signal processing. Although theBP learning has been a significant research area of artificial
neural network, it also has been known as an algorithm with apoor convergence rate. Many attempts have been made to thealgorithm to improve the performance on convergence speed
and learning efficiency. For example, there have been a lot ofreports on changing the learning rate and also the numberof neurons in the hidden layer but this will lead to slight
improvement only [3]-[5]. Not many studies have been madeon modifying the algorithm structure in order to improve thelearning performance.
On the other hand, chaos has gained much attention andsome applications in neural network over this recent years.
There have been many reports on the good performance of
Hopfield neural network when chaos is inputted to the neurons
as noise [6]. By computer simulations, it has been confirmedthat chaotic noise is effective for solving quadratic assigment
problem and gains better performance to escape out local
minima than random noise. The authors [7] also have proposed
the feedforward neural network with chaotically oscillating
gradient of the sigmoid function and proved that the proposed
network can find good solutions in early time. Hence, weconsider that various features of chaos can give a good effect inneural network. Network performance can be improved whenchaotic noise is applied to neural networks.
In this study, we conduct a further investigation by simulating our proposed network to learn sine wave function
with different frequencies. Different frequencies of sine wave
function may give different learning results and we analyze
the network performance.
II. BP LEARNING ALGORITHM
A. BP batch learning algorithm
In the standard backpropagation learning algorithm, theerrors of output neurons are backpropagated through thenetwork during training. This standard learning algorithm wasintroduced in [8]. The error signal of output neuron can bedefined by taking the difference between the target output andthe actual output. However, in this study, we use the batch
BP learning algorithm. The batch BP learning algorithm isexpressed by a formula similar to the standard BP learningalgorithm but the difference lies in timing of the weight update.
The weight update of the standard BP is performed after eachsingle input data, while for the batch BP the weight update is
performed after all input data has been processed. The totalerror E of the network is defined as follows;P∑pN∑1∑
E=Ep ={(tpi − opi )2 },
2
p=1
p=1
i=1

(1)

where P is the number of the input data, N is the number ofthe neurons in the output layer, tpi denotes the value of thedesired target data for the pth input data and opi denotes thevalue of the output data for the pth input data. The goal of thelearning is to set weights between all layers of the network sothat the total error E can be minimized. In order to minimize
the total error E, the weights are adjusted according to thefollowing equation:- 15 -

k−1,kk−1,kwi,j(m + 1) = wi,j(m) +P∑

k−1,k4p wi,j(m),(2)p=1
k−1,k4p wi,j(m) = −η∂Ep
k−1,k∂wi,j,(3)k−1,kwhere wi,jis the weight between ith neuron of the layerk − 1 and the jth neuron of the layer k, m is the learning timeand η is the learning rate. In this study, we add the inertia
term to Eq.(3) where ζ denotes the inertia rate.

k−1,k4p wi,j(m) = −η∂Epk−1,k∂wi,jk−1,k+ ζ4p wi,j(m − 1),(4)B. Chaotic BP learning algorithm
In our previous research, we have proposed a new modified
BP learning algorithm, namely chaotic BP learning algorithm.
This new algorithm can be expressed by a similar formulawith the standard BP algorithm but the different lies in weight
update process. Chaotic noise is added into weight updateprocess during error propagation. The weight update process
for chaotic BP learning algorithm can be shown as follows,which β denotes the noise amplitude.

III. S IMULATION RESULTS
In this section, we show the effectiveness of this chaotic
BP algorithm by testing it to a learning problem. Here, we
consider the feed forward neural network produce outputs
0.5 sin(2πf x + π/6) + 0.5 for input data x as one learning
example. The sampling range of the input data is [-1.0, 1.0].
We carried out the chaotic BP learning algorithm by using thefollowing parameters. The learning rate and the inertia rate are
fixed as η = 0.1 and ζ = 0.001 respectively. The initial valuesof the weights are given between -1.0 and 1.0 at random. The
learning iterations is set to 50000 and 8 neurons are preparedin the hidden layer (Fig. 2). For all simulation process,we use fixed value of chaos parameter (α=0.55) and noise
amplitude (β=0.01) to investigate the learning perforamancefor difference frequencies of sine wave function. The learning
example of sine wave function 0.5 sin(2πf x + π/6) + 0.5 isshown in Fig. 3 for f = 1.0[Hz].

∂Epk−1,k4p wi,j(m) = −η

k−1,k∂wi,jk−1,k+ζ4p wi,j(m − 1) + noisei,j (m),(5)noisei,j (m) = βi,j (m)(xi,j (m) − 0.5),(6)Chaos, xi,j (m) can be generated by various chaotic maps
namely logistic map, skew tent map, Benoulli shift map andmany others. Previously, we have investigated chaos featuresusing logistic map and confirmed that our improved algorithm
can give better convergence rate. In this study, we use otherchaotic map which is skew tent map to generate chaos. Theskew tent map and an example of time series obtained byEq.(7) are shown in Fig. 1. α is the control parameter of chaos
and one of the most important parameters for this work.
xi,j (m)αxi,j (m + 1) =Fig. 2.

Network structure.

(0 ≤ xi,j (m) ≤ α)(7)

 xi,j (m)−1(α < xi,j (m) ≤ 1)

α−1

1

1

0.8x(m+1)
0.6

0.5x(m)0.4

0.2

0000.5x(m)

10200

400

600

800

1000time(a) Skew tent map.
Fig. 1.

(b) Time series.

Fig. 3.
.

Learning example of sine wave function 0.5 sin(2πx + π/6) + 0.5

Skew tent map.

It should be noted that although there maybe more suitablevalues of these parameters, it is difficult to set them theoret-- 16 -

3.0. If f [Hz] is set to 0.5, there is no improvement at all
between our proposed network and conventional BP algorithm.
If frequency value is too large, the learning performance
become not stable and cannot find a good solution in earlytime. We can see the bad learning results if f [Hz] is set
to 5.0. The more difficult a learning function is, the more
complicated the learning process will be. Hence, we show theresult of all frequency values for proposed network-3 (Fig. 6).
We understand that a large value of frequency need more time
to solve the learning problem compared to small frequency
values.
Fig. 4.

Learning example of sine wave function 0.5 sin(2πx + π/6) + 0.5.

ically. Hence, we use the same parameters during all process
to make it easier to analyze the simulation results.
A. Noise added to different weight update positionsFirst, we investigate the learning efficacy by adding chaotic
noise into different position of weight update. As we may
know, in BP learning algorithm, the main purpose of weight
update is to reduce the error value between output and desired
target. Here, we choose one sine wave function with fixed
frequency (f = 1.0[Hz]) to see the effectiveness of noise
adding into different position of weight update. We add
chaotic noise into three different positions of weight update; a) Proposed network-1 (from input layer to hidden layeronly), b) Proposed network-2 (from hidden layer to outputlayer only) and c) Proposed network-3 (both of them). We
also compare the learning performance with the standard BP
algorithm which there is no chaotic noise application at all(conventional network). Figure 4 show the result performanceof all proposed and conventional network for sine wave
function learning example. The horizontal axis is iteration time
and the vertical axis is error value.
From this figure, we confirm that our proposed network
which chaotic noise is added into the weight update gains
better performance than the conventional network when chaosparameter of α is set to 0.55 and noise amplitude β is fixed
as 0.01. The addition of chaotic noise during weight update
can help the learning process to find a good solution inearly time compared to the standard BP algorithm but theimprovement rate is very small. Hence, we investigate thelearning performance if different frequencies of sine wave
function are used. Different frequencies of sine wave function
may give different learning results and we analyze the network
performance.
B. Different frequencies of sine wave function
We observe the network performance by simulating it tovarious sine wave function with different frequencies. Figure
5 show the learning performance when f [Hz] is set to 0.5,1.0, 2.0, 3.0, 4.0 and 5.0 respectively.
From these results, we can see that our proposed network
give better improvement rate when f [Hz] is set to 2.0 andFig. 6.
Learning performance of sine wave function with different
frequencies for proposed network-3.

IV. C ONCLUSIONSIn this study, we have conducted a further investigation of
our proposed network to learn sine wave function with different frequencies. From simulation results, we confirmed thatour proposed network can improve the learning performancecompared to the standard BP algorithm. We also discoveredthat learning process will take a longer time if more difficult
learning function is used.
R EFERENCES[1] A. Azamimi, Y. Uwate and Y. Nishio, “Good Learning Performance of
Backpropagation Algorithm with Chaotic Noise Features,” Proceedingsof SSJW”08,pp. 36-38, Aug. 2008.
[2] A. Azamimi, Y. Uwate and Y. Nishio, “An Analysis of Chaotic Noise
Injected to Backpropagation Algorithm in Feedforward Neural Network,”
Proceedings of IWVCC’08, pp. 70-73, Nov 2008.
[3] A. Sperduti and A. Starita, “Speed up learning and network optimization
with extended back-propagation,” Neural Networks, vol.6, pp.365-383,
1993.
[4] A. Miniani and R.D. Williams, “Acceleration of backpropagation throughlearning rate and momentum adaption,” Proceedings of IJCNN’90,SanDiego,CA, pp.676-679, 1990.
[5] R.A. Jacobs, “Increased rates of convergence through learning rateadaption,” Neural Networks, vol.1, pp.295-308, 1988.
[6] T. Ueta, Y. Nishio and T. Kawabe, “Comparison between Chaotic Noiseand Burst Noise on Solving Ability of Hopfield Neural Networks,”
Proceedings of NOLTA’97, vol.1, pp.409-412, 1997.
[7] Y. Uwate and Y. Nishio, “Chaotically Oscillating Sigmoid Function inFeedforward Neural Network,” Proceedings of NOLTA’06, pp.215-218,
2006.
[8] D.E. Rumelhart, G.E. Hinton and R.J. Williams, “Learning Internal
Representations by Error Propagation,” Parallel Distributed Processing,vol.1, MIT Press, MA, pp.318-362, 1986.

- 17 -

Fig. 5.

(a) f = 0.5 [Hz].

(b) f = 1.0 [Hz].

(c) f = 2.0 [Hz].

(d) f = 3.0 [Hz].

(e) f = 4.0 [Hz].

(f) f = 5.0 [Hz].

Learning performance of proposed and conventional network for sine wave function with different frequencies.

- 18 -