Feedforward Neural Networks with Fluctuation of Structure
for Chaotic Time Series Learning
Yoko Uwate† , Yoshifumi Nishio‡ and Ruedi Stoop††Institute of Neuroinfomatics, University / ETH Zurich,
Winterthurerstrasse 190, 8057 Zurich, Switzerland,
email: {yu001, ruedi}@ini.phys.ethz.ch‡Dept. of Electrical and Electronic Engineering, Tokushima University
2-1 Minami-Josanjima, Tokushima, Japan
Email: {uwate, nishio}@ee.tokushima-u.ac.jp
Abstract
The fluctuation observed in the real brain has attracted
attentions of many researchers. We consider that the fluctuation plays an important role for the higher brain function.
In this study, the neural network with fluctuation of structure is investigated. By computer simulations, we confirm
that the neural network with fluctuation of structure gains
better performance than the conventional neural network
for generalization ability.
1. Introduction
Studies on the human brain have been carried out actively on various levels. Many modelings of the humanbrain with the visual or the audio sensation are reported [1].
Recently, the fluctuation observed in the real brain has attracted attentions of many researchers. The fluctuation in
the brain is one of the living evidence, and the amplitudeof the fluctuation decreases or disappears when human becomes sick. It can be said that the fluctuation has somerelationship to the higher brain functions. We consider that
it is very important to apply this fluctuation factors to artificial neural network model.
In our previous research, we have investigated the abilityof the feedforward neural network when the shape of thesigmoid function of neurons is changed according to thelogistic map equation [2]. We have compared its effect withthe simulated annealing and have confirmed that changesometimes led the state of the network to better solutions.
We consider that the neural network with fluctuation ofstructure has many possibilities for the information processing. We assume that the real brain operates well when
the neural circuits are shaken by some kinds of influences.
In this study, the learning ability of the network with fluctuation of structure is investigated. Two methods giving fluctuation are proposed to investigate the most efficient method providing fluctuations; coherent fluctuation
method and noncoherent fluctuation method. For comparison, we investigate the network shaken by additional noise
signals. By computer simulations, we confirm that the pro-

posed network with fluctuation of structure can find better
solutions than the network with additional noise signals andthe conventional network for generalization ability.
2. Neural Network with Fluctuation of Structure
The standard BP learning algorithm was introduced in[3]. The effectiveness of the BP learning has been confirmed in pattern recognition, system control, signal processing and so on [4]. In this study, the learning abilityand generalization ability of the network with fluctuationof structure are investigated by using the batch BP learningalgorithm. .
The sigmoid function has an effect on modifying connection weights and it is very important for BP learning. Thisfunction is an S shaped monotonic increasing function thathas the general form as following equation:f (x) =1
1 + e−εx(1)where ε is a constant that determines the steepness of theS shaped curve. Some curves of the function for different
values of ε are illustrated in Fig. 1.
We apply the sequence generated by the uniform randomfunction to ε of the sigmoid function after the following
linear transform to set the average as 1.0 and control theamplitude.
ε(t) = A(random() − 0.5) + 1.0(2)where random() means the function producing uniform
random value from 0.0 to 1.0 and, A corresponds to therange of ε. In this simulation, we set from A = 0.02 to 0.6.
Next, two fluctuation methods are proposed as follows;
2.1. Coherent Fluctuation Method
The gradient of the sigmoid function fluctuates with acoherent value for all neurons in the hidden layer at every
updating.

- 59 -3. Neural Network with Additional Noise Signals

1For comparison, we investigate the network shaken byadditional noise signals. We consider that the uniform
noise (generating value form 0.0 to 1.0) is injected to neurons in the hidden layer. Figure 3 shows a conceptual neuron model for this neural network, where β limits the amplitude of the injected signals.

0.8
0.6
0.5
0.4
0.2

ε:small
ε=0.2

ε=0.5
ε=1.0

0-15-10-5ε:large
ε=2.0
05

10x11(t)15xx12 (t)Figure 1: Sigmoid function and fluctuation ε.

w im;11w im;12.
.
. w im;1n.
βx1n(t)2.2. Noncoherent Fluctuation Method
The gradient of the sigmoid function fluctuates with noncoherent different values for all neurons in the hidden layer
at every updating by giving different initial conditions tothe logistic map.
A conceptual figure of the two fluctuation methods areshown in Fig. 2.

x (t+1)fz(t)Noise signal-1

θFigure 3: A neuron model with additional noise signals.

4. Simulated Results
We consider that the neural network with fluctuation ofstructure has a generalization ability as well as a learning
ability by virtue of its inefficient learning process. In thissection, we investigate the learning ability and the generalization ability of our proposed networks. We consider thelearning of the structure of the skew tent map by training
the network to output the same time series as the input timeseries produced by the skew tent map.
The skew tent map and an example of time series areshown in Fig. 4. The length of chaotic time series is set to
10 and the number of learning patterns is set to 10. Whenthe network learns 10 lengths of time series, 10 nodes areprepared in the input and the output layers. Each data isinputted to each node in the input layer. We carried out
the BP learning by using the following parameters. The
parameter of the inertia rate is fixed as η = 0.05 and theinitial values of the weights are given between −1.0 and
1.0 at random. The learning time is set to 10000, and 12
neurons are prepared in the hidden layer. The gradient of
the sigmoid function of the conventional network is fixedas ε = 1.0.

1stneuron2nd
neuron3rd
neuronNthneuron
iteration time(a) Coherent fluctuation method.
1stneuron

2ndneuron

3rdneuron4.1. Learning Ability
Nthneuron
iteration time(b) Noncoherent fluctuation method.

Figure 2: A conceptual figure of two fluctuation methods.

First, we investigate the learning efficiency as the average of the total error between the output and the desired
target, when the network structure of the hidden layer ischanged. The “Average Error Eave ” for this learning example is defined by the following equation.
 NP
1 1 X
1 X
2
.
(3)
(t pi − o pi ) 
Eave =2PN
p=1- 60 -i=1

0.5

10.0010.8

0.00090.60.0008

Eave for learning data

Chaotic time series

1x n+10.4

0.2

000.5 a=0.55

0xn

1

01

2

34

56

789time(a) Skew tent map.

(b) Time series.

Noise signal0.00070.00060.00050.00040.00030.0002

Figure 4: Skew tent map.

00.02

The simulated result of networks with fluctuation ofstructure is shown in Fig. 5. When the value of fluctuation A is 0.02, the methods giving fluctuation and the conventional network obtain similar performance. While thefluctuation methods becomes worse by increasing A. We
compare the performance between the coherent and noncoherent fluctuation methods. When A is small value, bothfluctuation methods show similar performance of Eave . The
noncoherent fluctuation method can keep the small error(Eave ) for learning data with increasing A.
Figure 6 shows the simulated result of network withnoise signals. The horizontal axis is β which means theamplitude of injected noise and the vertical axis is Eave forlearning data. From this figure, we can confirm the noise
signals method shows similar performance to the conventional network when β is set to 0.02. However, the performance of noise signals method is getting worse by increasing β value.
0.0002

Eave for learning data

0.00018

Coherent fluctuation

0.000160.000140.00012

Noncoherent fluctuation

0.0001
8e-05

Conventional network

6e-05
4e-05
2e-05

00.1

0.2

0.3

0.4

0.5

0.6

0.7

Conventional network

0.0001

0.8

0.9

1

A (range of fluctuation)

Figure 5: Network performance Eave by using coherent andnoncoherent fluctuation methods for learning data.
The simulated results of the fluctuation methods aresummarized in Tab. 1. In this table, the average of 10 Eavefor different initial conditions of the weights between alllayers is shown. From this table, we can confirm that Eaveof the proposed network with the fluctuation methods andthe conventional network is similar. We consider that theperformance of fluctuation methods are not so difference tothe conventional network for learning ability.

0.04

0.06

0.08

0.1

0.12

0.14

0.16

β: amplitude of injected noise

0.18

0.2

Figure 6: Network performance Eave by using noise signals
methods for learning data.
4.2. Generalization Ability
Next, we investigate a generalization ability of the proposed neural networks. After the above learning of 10 patterns of the time series, we input an unknown chaotic timeseries generated the same skew tent map as an input pattern.
The simulation result for unknown chaotic time series
after learning of the network with fluctuation of structureis shown in Fig. 7. The horizontal axis is the range offluctuation A and the vertical axis is Eave for 10 different
unknown input data. From this figure, the networks withfluctuation of structure gain better performance than theconventional network in every range of A. We compare
the performance between the coherent and the noncoherent
fluctuation methods. When A is small value, the coherent
fluctuation method is better than the noncoherent fluctuation method. However, the coherent and the noncoherent
fluctuation methods are getting similar performance withincreasing A.
Figure 8 shows the simulated result of network withnoise signals. From this figure, we can confirm the noise
signals method obtains almost similar performance to theconventional network.
The simulated results of the fluctuation methods for unknown data are summarized in Tab. 2. In the table, theaverage of 10 Eave for different initial conditions of theweights between all layers is shown. From this table, Eaveof the proposed network with the fluctuation of structure
are small. However, the noise signal method and the conventional network do not operate well, because the average
error Eave of the network is large. We can see that the network with fluctuation of structure gain better performanceon generalization ability than the noise signal method andthe conventional network.
From these results, we consider that the fluctuation ofstructure are very important role to learn some characteristics or features of the given data.
Before concluding this paper, we have to say that the net-

- 61 -

Table 1: Learning ability for learning data.
Coherent
fluctuation
0.000038

Eave

Noncoherent
fluctuation
0.000038

Noisesignals
0.000057

Conventionalnetwork
0.000039

Table 2: Generalization ability for unknown data.
Coherent
fluctuation
0.017382Eave

Noncoherent
fluctuation
0.013418

Eave for unknown data

Conventionalnetwork
0.025735

work does not learn to output the same data as input data
directly. In our past study, we have investigated the performance of the network when a random data is inputted as
unknown data after the learning. We confirmed the network
does not output the same data as input data directly [8].

0.03Conventional network

0.028

Noise
signals
0.025440

0.026
0.024
0.022

5. Conclusions

Coherent fluctuation

0.02
0.018
0.016
0.014

Noncoherent fluctuation

0.012
0.01

00.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.91

A (range of fluctuation)

Figure 7: Network performance Eave by using coherent andnoncoherent fluctuation methods for unknown data.

In this study, we investigated learning ability and generalization ability of neural network with fluctuation of structure for back propagation learning. By computer simulations, we confirmed that the proposed network with fluctuation can produce the time series for unknown input data.
We can see that the network with fluctuation of structure
operates more effectively than the network with additional
noise signals for generalization ability. Furthermore, thenoncoherent fluctuation gains better performance than thecoherent fluctuation for unknown input data. The detailed
investigation of the effect of different fluctuation methods
is our future work.
References

Eave for unknown data

0.0265

Conventional network

0.026

0.0255

Noise signal
0.025

0.02450.024

0.0235
0.02

0.04

0.06

0.08

0.1

0.12

0.14

0.16

β: amplitude of injected noise

0.18

0.2

Figure 8: Network performance Eave by using noise signals
methods for unknown data.

- 62 -[1] R. P. Broussard, S. K. Rogers, M. E. Oxley and G. L. Tarr, “Physiologically Motivated Image Fusion for Object Detection Usinga Pulse Coupled Neural Network,” IEEE Transactions on NeuralNetworks, 10 (3), pp. 554-563, 1999.
[2] Y. Uwate and Y. Nishio, “Chaotically Oscillating Sigmoid Function
in Feedforward Neural Network,” Proc. NOLTA’06, pp. 215-218,
Sep. 2006.
[3] D.E. Rumelhart, G.E. Hinton and R.J. Williams, “Learning Internal
Representations by Error Propagation,” Parallel Distributed Processing, vol. 1, MIT Press, MA, pp. 318-362, 1986.
[4] Y. LeCun, B. Boser, J.S. Denker, et al, “Handwritten Digit Recognition with a Back-Propagation Network,” Advances in Neural Information Processing Systems, vol. 2, MIT Press, MA, 1990.
[5] T. Ueta, Y. Nishio and T. Kawabe, “Comparison between Chaotic
Noise and Burst Noise on Solving Ability of Hopfield Neural Networks,” Proc. NOLTA’97, vol. 1, pp. 409-412, Nov. 1997.
[6] Y. Uwate, Y. Nishio and A. Ushida, ”Markov Chain Modeling ofIntermittency Chaos and its Application to Hopfield NN,” IEICETransactions on Fundamentals, vol. E87-A, no. 4, pp. 774-779,Apr. 2004.
[7] Y. Uwate, Y. Nishio, T. Ueta, T. Kawabe and T. Ikeguchi, “Performance of Chaos and Burst Noise Injected to the Hopfield NN for
Quadratic Assignment Problems,” IEICE Transactions on Fundamentals, vol. E87-A, no. 4, pp. 937-943, Apr. 2004.
[8] Y. Uwate and Y. Nishio, “Performance of Affordable Neural Network for Back Propagation Learning,” IEICE Transactions on Fundamentals, vol. E89-A, no. 9, pp. 2374-2380, Sep. 2006.