2005 RISP International Workshop on Nonlinear
Circuit and Signal Processing (NCSP'05)
Hawaii, USA, Mar. 4-6, 2005.

Back Propagation Learning of an Affordable Neural Network for Chaotic Time Series
Yoko Uwate, Natsumi Dake and Yoshifumi Nishio
Dept. of Electrical and Electronics Engineering,
Tokushima University
2-1 Minami-Josanjima, Tokushima, Japan
Phone:+81-88-656-7470
FAX:+81-88-656-7471
Email: {uwate, dakera55, nishio}@ee.tokushima-u.ac.jp
1. Introduction

activated. The network model with the affordable neurons isshown in Fig. 1.

In Japan, cramming of too much knowledge into students
was criticized and more relaxed education policy has been
introduced to develop the individuality of each student. Ifstudents can afford to study carefully, creativity of the individual is fostered. We consider that it is very important to pay
attention to “Affordable” concept in the field of engineering.
In our previous research, we have proposed a new network
structure with chaotically-selected affordable neurons in thehidden layer of the feedforward neural network for more efficient BP learning [1]. By computer simulations, the proposed
neural network has been confirmed to gain better performancefor the BP learning on both convergence speed and learning
efficiency, when we set the feedforward neural network producing outputs x2 for inputs data x as a learning example.
However, only the results on one learning example can notconclude that the affordable neural network has an excellent
ability for convergence speed and learning efficiency.
In this study, we investigate the performance of this neuralnetwork with chaotically-selected affordable neurons for unknown data which is chaotic time series generated by a skew
tent map. Further, in order to confirm the effectiveness ofthe chaotic selection of the affordable neurons, we investigate performance of the network with affordable neurons selected regularly and at random. Computer simulated results
show that the affordable neural network exhibits a good performance for the unknown input data.
2. Affordable Neural Network
We propose the network structure with chaotically-selectedaffordable neurons in the hidden layer of the feedforward neural network for more efficient BP learning. We introduce affordable neurons to reflect a function of the brain. The extra
neurons in the hidden layer are prepared in advance. During the BP learning, all of the neurons in the hidden layer are
not used at every updating. Namely, some of the neurons are
selected for the learning and the rest of the neurons are de-Chaos

Chaos
Chaos
Chaos
Chaos

Input layer

Chaos

Output layer

Chaos

Chaos

Affordable
neuronHidden layer

Figure 1: Network model with affordable neurons.

2.1. Chaotic Selection
For the proposed network, some of the neurons have to be
selected at every updating of the BP learning. The authors
have investigated the performance of the Hopfield neuralnetwork solving combinatorial optimization problems whenchaos is inputted to the neurons as noise [2]-[4]. By computer simulations, chaotic noise has been confirmed to gain
better performance to escape out of local minima than random noise. Hence, we consider that various features of chaos
are effective for neural networks.
In this study, we use the skew tent map as a simple chaotic
map to realize chaotic selection of operating neurons in thehidden layer. We prepare the skew tent maps with different
initial values, whose number is the same as the number ofthe neurons in the hidden layer, and each skew tent map iscorresponded to each neuron. At the every updating of the- 139 -BP learning, the skew tent maps are also updated and theirvalues are referred. We select a certain number of the neurons
in the order of the values of the skew tent maps. Hence, theselection is made in a chaotic manner. Note that chaos is not
inputted to the neural network directly, but is used only forthe selection of the operating neurons. The skew tent map isdefined by the following equation and the map is shown inFig. 2.

1x n+10.5

0.5 a=0.55

xn1

3.1. Conventional network

Figure 2: Skew tent map.

xn+1 =

xn
 0.5 + a (0 ≤ xn ≤ a)−xn + 1(a < xn ≤ 1)
1−a(1)

2.2. Regular and Random Selection

In order to confirm the effectiveness of the chaotic selection of the affordable neurons, we also investigate the performance of the BP learning of the network with affordable
neurons selected other non-chaotic methods, namely regularmethod and random method. For the regular method, the operating neurons in the hidden layer are selected in order. For
the random method, the operating neurons in the hidden layer
are selected completely at random.
3. Simulated Results

In order to make clear the reason why affordable neuralnetwork gains good performance for unknown input data, weinvestigate the performance of the conventional neural network without affordable neurons. The simulation result ofconventional network is shown in Fig. 3. The horizontal axis
is time and vertical axis is unknown chaotic time series generated by the skew tent map. From this figure, we can seethat the performance of conventional neural network does notwork well for unknown input data. Because, the error of output for input data is 0.1023.
1

Chaotic time series by skew tent map

00

In this study, after learning of the affordable neural network, we input an unknown chaotic time series generated byskew tent map as another learning data. We investigate theaffordable neural network producing the same output as theinput data. Further, we compare three cases “chaos”, “regular” and “random” method as selection of the affordable neurons. We prepare 20 neurons in hidden layer and the numberof affordable neurons is changed as 0 to 10. If the numberof affordable neurons is 0, the network has a normal (conventional) hidden layer structure. However, if the number ofaffordable neurons is 10, only 10 neurons of 20 neurons in
hidden layer are operated at every update time.
Next, we explain about the learning of the network. We
consider the feedforward neural network producing the samechaotic time series as the inputed chaotic time series generated by a skew tent map as one learning example. The lengthof chaotic time series is set to 10 and the number of learning
data is set to 100. We carried out the BP learning by using following parameters. The parameter of the inertia rate is fixedas η = 0.05 and initial values of the weights are given between −1.0 and 1.0 at random. The learning time is set to20000.

0.8

0.6

0.4

0.2
The standard BP learning algorithm was introduced in [5].
Input
Output
The BP is the most common learning algorithm for feedforward neural networks. In this study, we use the batch BP00123456789time
learning algorithm. The batch BP learning algorithm is expressed by similar formula of the standard BP learning algorithm. The difference lies in the timing of the update ofthe weight. The update of the standard BP is performed after Figure 3: Input and output data of conventional network (ereach single input data, while the update of the batch BP is ror: 0.1023).
performed after all different input data.

- 140 -

Figure 4(a) shows the output data for unknown input datausing by chaos selected affordable neural network. The horizontal axis is time and vertical axis is unknown chaotic timeseries generated by the skew tent map. From this figure, weconsider that the chaos selected affordable neural network canoutput the same unknown input data and the error is 0.0059.
Next, the state of selected affordable neurons is shown inFig. 4(b). The horizontal axis is time and vertical axis isneuron number in hidden layer. The plotted marks mean toselected as affordable neuron at each updating. We can seethat the affordable neurons are selected chaotically.

Chaotic time series by skew tent map

10.8

this figure, we consider that the regular selected affordable
neural network gains better performance than the conventional network. However, this regular selection method does
not to be compared with chaos selection method. Because theerror of the network for output is 0.0129. Next, the state of
selected affordable neurons is shown in Fig. 5(b). We can seethat the affordable neurons are selected regularly.
1

Chaotic time series by skew tent map

3.2. Affordable neural network using CHAOS selection

0.6

0.4

0.2

Input

0.6Output
00.40123456789time(a) Input and output data (error: 0.0129).
0.2Input
20Output
01234time56789Neuron number in hidden layer0(a) Input and output data (error: 0.0059).
20Neuron number in hidden layer

0.8151015105050100time150

200(b) Selected affordable neurons.

500050100time150

Figure 5: Regular selection (the number of affordable neurons: 2).

200(b) Selected affordable neurons.
Figure 4: Chaos selection (the number of affordable neurons:2).

3.4. Affordable neural network using RANDOM selection

Figure 6(a) shows the output data for unknown input datausing by random selected affordable neural network. From
3.3. Affordable neural network using REGULAR selec- this figure, we consider that the regular selected affordable
neural network almost can output the same unknown input
tion
data and the error is 0.0098. Next, the state of selected afFigure 5(a) shows the output data for unknown input data fordable neurons is shown in Fig. 6(b). We can see that theusing by regular selected affordable neural network. From affordable neurons are selected at random.

- 141 -0.4
Chaos

0.35
0.8

Random

0.25

0.6

0.4

0.2
0.15
0.1

0.2

Input
0.05

Output0Regular

0.3

Error

Chaotic time series by skew tent map

101

2

3

4

time

56

7

8

010

98

6420

Number of affordable neurons in hidden layer(a) Input and output data (error: 0.0098).
Figure 7: Error for unknown data.
20

Neuron number in hidden layer

4. Conclusions
15In this study, we applied this neural network withchaotically-selected affordable neurons to learning unknown
chaotic time series generated by skew tent map. Computer
simulated results showed that the affordable neural network
exhibits a good performance for the unknown input data. Further, we confirmed that the chaotic and random selection ofaffordable neurons was important to obtain the good performance. These results of the affordable neural network areused the noise reduction in chaos communication [6].

1050050

100time150200(b) Selected affordable neurons.

References

Figure 6: Random selection (the number of affordable neurons: 2).

[1] Y. Uwate and Y. Nishio, “Back Propagation Learning of Neural Networks with Chaotically-Selected Affordable Neurons,”
Proc. ISCAS’05, May 2005 (to appear).

3.5. Comparison of three selection methods[2] T. Ueta, Y. Nishio and T. Kawabe, “Comparison between
Chaotic Noise and Burst Noise on Solving Ability of Hopfield
Neural Networks,” Proc. NOLTA’97, vol. 1, pp. 409-412, Nov.
1997.

Finally, we investigate the performance of three selection
methods, when the network structure of the hidden layer ischanged. The simulation result is shown in Fig. 7. The horizontal axis is the number of affordable neurons and the vertical axis is the error for the unknown input data. In this figure, we confirm that the neural network with some affordable
neurons gains the better performance than the network without affordable neurons in hidden layer. However, when thenumber of the affordable neurons exceeds 4, the performanceof the network gradually becomes worse because the number of operating neurons decrease and the networks becomesluggish. Further, we confirm that the chaos selection and therandom selection gain better performances than the regularselection.

[3] Y. Uwate, Y. Nishio and A. Ushida, ”Markov Chain Modelingof Intermittency Chaos and its Application to Hopfield NN,”IEICE Transactions on Fundamentals, vol. E87-A, no. 4, pp.
774-779, Apr. 2004.
[4] Y. Uwate, Y. Nishio, T. Ueta, T. Kawabe and T. Ikeguchi, “Performance of Chaos and Burst Noise Injected to the Hopfield
NN for Quadratic Assignment Problems,” IEICE Transactions
on Fundamentals, vol. E87-A, no. 4, pp. 937-943, Apr. 2004.
[5] D.E. Rumelhart, G.E. Hinton and R.J. Williams, “Learning
Internal Representations by Error Propagation,” Parallel Distributed Processing, vol. 1, MIT Press, MA, pp. 318-362, 1986.
[6] N. Dake, Y. Uwate, Y. Nishio, T. Yamazato, “Research on
Clutter Reduction Methods of a UWB Chaotic Radar,” Proc.
NCSP’05, Mar. 2005 (to appear).

- 142 -