NOLTA 2006
11-14 september
Bologna - ITALY

Chaotically Oscillating Sigmoid Function
in Feedforward Neural Network
Yoko Uwate and Yoshifumi Nishio
Dept. of Electrical and Electronics Engineering,
Tokushima University, 2-1 Minami-Josanjima, Tokushima, Japan,
Phone: +81-88-656-7470, Fax: +81-88-656-7471,
Email: {uwate, nishio}@ee.tokushima-u.ac.jp.
Abstract—In this study, we propose the feedforward neural network with chaotically oscillating sigmoid function. By computer simulations, we confirm
that the proposed neural network can find good solutions in early time of the back propagation learning
process.
1. Introduction
Recently, studies on the human brain have been carried out actively on various levels. Many modelingsof the human brain with the visual or the audio sensation are reported [1]-[3] due to development of thebrain researches. However, the investigation of modeling of higher functions in the human brain is justgetting started. We consider that it is very important to apply these high functional mechanisms of thehuman brain to novel artificial neural networks.
Back Propagation (BP) learning [4] is one of engineering applications of artificial neural networks. TheBP learning operates with a feedforward neural network which is composed of an input layer, a hidden
layer and an output layer, and the effectiveness of theBP learning has been confirmed in pattern recognition, system control, signal processing, and so on [5][7]. The BP learning process requires that the inputoutput functions are bounded and differentiable functions. One of the most commonly used functions satisfying these requirements is the sigmoid function. The
nonlinearity of the sigmoid function has an effect on
modifying connection weights and it is very importantfor BP learning.
We consider that neurons in the human brain do notalways output the same output for the same input. Inorder to reflect this idea to the feedforward neural networks, in this study, we propose the feedforward neuralnetwork with chaotically oscillating gradient of the sigmoid function. In order to confirm the effectiveness ofthe chaotically oscillating sigmoid function, we carryout computer simulations using other shaking methods. Further, we compare the proposed network to theSimulated Annealing (SA) method. By computer simulations, we confirm that the proposed network withchaotically oscillating sigmoid function can find goodsolutions in early time of the BP learning process.
2. Chaotically Oscillating Sigmoid Function
The BP learning process requires that the inputoutput functions are bounded and differentiable functions. One of the most commonly used functions satisfying these requirements is the sigmoid function. Thisfunction is an S shaped monotonic increasing functionthat has the general form as following equation:f (x) =1
1 + e−εx(1)where ε is a constant that determines the steepness of
the S shaped curve. Some curves of the function fordifferent values of ε are illustrated in Fig. 1.
10.8

0.60.5ε:small

0.4

ε=0.2

0.20-15-10ε:large
ε=0.5
ε=1.0 ε=2.0-5051015xFigure 1: Sigmoid function.
The nonlinearity of the sigmoid function has an effect on modifying connection weights and it is very important for BP learning. We propose the feedforward
neural network with chaotically oscillating gradient (ε)
of the sigmoid function for BP learning.
The authors have investigated the performance ofthe Hopfield neural network solving combinatorial optimization problems when chaos is inputted to theneurons as noise [8]-[10]. By computer simulations,chaotic noise has been confirmed to gain better performance to escape out of local minima than random
noise. Hence, we consider that various features ofchaos are effective for neural networks.

215

The logistic map is used to shake ε of the sigmoid
function chaotically:ε̂(t + 1) = αε̂(t)(1 − ε̂(t)).

equation:
k−1,kk−1,kwi,j(m + 1) = wi,j(m) +(2)Varying the parameter α, Eq. (2) behaves chaotically
via a period-doubling cascade. Further, it is wellknown that the map produces intermittent bursts justbefore periodic-windows appear. We apply the sequence generated by the logistic map to the sigmoid
function after the following linear transform to set thestandard as 1.0 and control the amplitude.
ε(t) = 2A(ε̂(t) − 1) + 1

k−1,k∆p wi,j(m),

p=1

k−1,k
∆p wi,j(m) = −η∂Ep,

k−1,k∂wi,j(5)k−1,kwhere wi,jis the weight between the ith neuron of
the layer k − 1 and the jth neuron of the layer k, m
is the learning time, and η is a proportionality factor
known as the learning rate. In this study, we add tothe second line of Eq.(3) an inertia term, leading to(3)where A corresponds to the range of ε. One exampleof A = 0.5 is shown in Fig. 2.

PXk−1,k∆p wi,j(m) = −η∂Epk−1,k∂wi,jk−1,k+ζ∆p wi,j(m−1), (6)where ζ denotes the inertia rate.

1.5

4. Simulated Results

1.4
1.3

We consider the feedforward neural network producing outputs x2 for inputs data x as one learning example. The sampling range of the input data is [−1.0, 1.0]
and the step size of the input data is set to be 0.01.
We carried out the BP learning by using the following
parameters. The learning rate and the inertia rate are
fixed as η = 0.2 and ζ = 0.02, respectively. The initialvalues of the weights are given between −1.0 and 1.0
at random. The learning time is set to 10000, and the8 neurons are prepared in the hidden layer. The network structure using this study and learning example
are shown in Fig. 3.

epsilon

1.2
1.1
1
0.9
0.8
0.7
0.6

0

2000

4000

6000

8000

10000

time

Figure 2: Oscillating ε by logistic map.

3. BP Learning Algorithm
The standard BP learning algorithm was introduced
in [4]. The BP is the most common learning algorithm
for feedforward neural networks. In this study, we use
the batch BP learning algorithm. The batch BP learning algorithm is expressed by a formula similar to thestandard BP learning algorithm. The difference lies inthe timing of the weight. The update of the standard
BP is performed after each single input data, while
for the batch BP the update is performed after all input data has been processed. The total error E of thenetwork is defined as( N)PPX
X1X
2E=Ep =(tpi − opi ) ) ,(4)
2 i=1
p=1
p=1
where P is the number of the input data, N is thenumber of the neurons in the output layer, tpi denotes
the value of the desired target data for the pth input
data, and opi denotes the value of the output data
for the pth input data. The goal of the learning isto set weights between all layers of the network so as
to minimize the total error E. In order to minimize
E, the weights are adjusted according to the following

10.8
0.6

Input

x2

Output

x

0.4

x2

0.2
0-1

(a) Network structure.

-0.5

0x

0.5

1

(b) Learning example.

Figure 3: Network structure and learning example.

4.1. Performance of Learning Process
We investigate the learning efficiency as the total
error between the output and the desired target. We
define “Average Error Eave ” by the following equation
as mean square error.
PEave =1 XP p=1½

1(tp − op )22¾(7)The bifurcation parameter of the logistic map is set
to α = 3.8274 generating intermittency chaos. The216

gradient of the sigmoid function of the conventional
network is fixed as ε = 1.0. Figure 4 shows one example of the simulation results when the amplitude of
ε of the sigmoid functions are changed. The horizontal axis is iteration time and the vertical axis is Eave .
This figure shows three learning curve of the proposed
network and the conventional network, respectively,
when the initial conditions of the connection weights
are changed. From this figure, we can confirm thatthe proposed network with the chaotically oscillating
sigmoid function gains better performance than theconventional network when the amplitude of ε is set
to 0.5 (Fig. 4(a)). The learning curve of the proposed
network converges oscillatory. On the other hand, theproposed network shows similar or weak performanceto the conventional network when the amplitude of εare set to 0.2 and 0.8 (Fig. 4(b) and (c)). We consider
that the proposed network gains good performance byusing appropriate amplitude of ε of the chaotically oscillating sigmoid function.

1
Eave
0.1
Conventional network (3)Conventional network (2)0.01

Conventional network (1)0.001

0.0001

Proposed network (1)Proposed network (2)Proposed network (3)0.00001

02000

4000

6000

8000

10000

time

(a) A=0.5.
1
Eave
0.1

Conventional network (3)Conventional network (1)0.01

Conventional network (2)0.001
Proposed network (1)Proposed network (2)0.0001

Proposed network (3)4.2. Comparison of Shaking Methods
0.00001

02000

4000

6000

8000

10000

time

In this section, in order to confirm the effectivenessof the chaotically oscillating sigmoid function, we carryout the computer simulations by other shaking methods; fully developed chaos and at random. The fully
developed chaos was realized by setting parameter ofthe logistic map (Eq. (2)) as α = 4.0000. The simulation result is shown in Fig. 5. In this figure, the resultsof the four cases “intermittency chaos,” “fully developed chaos”, “random” and “conventional network”
are shown.
We can confirm that the learning curve of the random method is more oscillatory than the intermittencychaos and the fully developed chaos. We consider thatthe chaotically oscillating ε is important to find goodsolution for BP learning.

(b) A=0.2.
1
Eave
Proposed network (3)Proposed network (2)0.1

Proposed network (1)
0.01

0.001
Conventional network (1)Conventional network (2)0.00010.00001

Conventional network (3)

02000

4000

6000

8000

10000

time

(c) A=0.8.
Figure 4: Learning curve for difference range of ε.

4.3. Efficient Learning
The proposed network with chaotically oscillating
sigmoid function finds better solutions than the conventional network. However, the learning curve of the
proposed network is oscillatory and do not converge.
We consider that the proposed network can converge
effectively, if oscillating ε stops on BP learning.
Figure 6 shows the simulated result when thestopped iteration time is set to 3000, 4000 and 5000.
After oscillating stop, the gradient is fixed as ε = 1.0.
From this figure, we can see that the proposed network converges to good solution by oscillating stop.
We consider that it is important to oscillate the gradient of the sigmoid function in early time of learning
process to find good solution.

1
EaveIntermittency chaos (A=0.5)
Fully developed chaos (A=0.5)
Random (A=0.5)
Conventional network

0.1

0.01

0.001

0.00010.00001

02000

4000

6000

8000

10000

time

Figure 5: Learning curve for difference shaking methods.

217

11
Eave

Eave

Stop time: 3000
Stop time: 4000
Stop time: 5000
Conventional network

0.1

0.01

0.01

0.001

0.001

0.00010.00010.00001

02000

4000

6000

8000

SA
Chaotically changed SA
Conventional network
Proposed network (Stop: 5000)

0.1

0.00001

10000

02000

time

4000

time

6000

8000

10000

Figure 8: Learning curve of SA method.

Figure 6: Learning curve for oscillating stop.

5. Conclusions
4.4. SA Method

1.6

1.6

1.4

1.4

1.2

1.2

epsilonepsilon

Simulated annealing (SA) is a generic probabilistic meta-algorithm for global optimization problems,
namely locating a good approximation to the global
optimum of a given function in a large search space.
SA can find a good solution by decreasing the gradientof the sigmoid function gradually. In this section, weinvestigate the performance of the SA method and theproposed network with the concept of SA. For comparison, the learning ability of the proposed networkand the conventional network are investigated. Thechanging ε in these networks are shown in Fig. 7. The
horizontal axis is time and the vertical axis is ε.

10.8References[1] R. P. Broussard, S. K. Rogers, M. E. Oxley and G. L. Tarr,“Physiologically Motivated Image Fusion for Object Detection Using a Pulse Coupled Neural Network,” IEEE Transactions on Neural Networks, 10 (3), pp. 554-563, 1999.
[2] R. Stoop and A. Kern, “Two-Tone Suppression and Combination Tone Generation as Computations Performed by theHopf Cochlea,” Phys. Rev. Lett., 93 (26) 268103, 2004.
[3] R. Stoop, W.-H. Steeb, J.C. Gallas and A. Kern, “Auditory
Two-Tone Suppression from a Subcritical Hopf Cochlea,”
Physica A, 351: pp. 175-183, 2005.
[4] D.E. Rumelhart, G.E. Hinton and R.J. Williams, “Learning
Internal Representations by Error Propagation,” Parallel
Distributed Processing, vol. 1, MIT Press, MA, pp. 318-362,1986.
[5] Y. LeCun, B. Boser, J.S. Denker, et al, “Handwritten Digit
Recognition with a Back-Propagation Network,” Advancesin Neural Information Processing Systems, vol. 2, MITPress, MA, 1990.
[6] M. Kawano, Y. Uno, M. Isobe and R. Suzuki, “A Hierarchical Neural Network Model for Voluntary Movement with
Application to Robotics,” IEEE Control Systems Magazine,
vol. 8, pp. 8-16, 1988.
[7] T.J. Sejnowski and C.R. Rosenberg, “Parallel Networks that
Learn to Pronounce English Text,” Complex Systems, vol.
1, pp. 145-168, 1987.
[8] T. Ueta, Y. Nishio and T. Kawabe, “Comparison between
Chaotic Noise and Burst Noise on Solving Ability of Hopfield Neural Networks,” Proc. NOLTA’97, vol. 1, pp. 409412, Nov. 1997.
[9] Y. Uwate, Y. Nishio and A. Ushida, ”Markov Chain Modeling of Intermittency Chaos and its Application to Hopfield
NN,” IEICE Transactions on Fundamentals, vol. E87-A,
no. 4, pp. 774-779, Apr. 2004.

10.8

0.6

0.4

In this study, we proposed the feedforward neural network with chaotically oscillating gradient ofthe sigmoid function. By computer simulations, weconfirmed that the feedforward neural network withchaotically oscillating sigmoid function can find goodsolutions in early time of the BP learning process.

0.602000

4000

6000

8000

0.4

10000

02000

4000

time

6000

8000

10000

time

(a) SA method.

(b) Proposed network with SA.

1.6
1.4
1.4

1.2

epsilonepsilon

1.2

10.8

1

0.8

0.6
0.6
0.4

02000

4000

6000

8000

time

(c) Proposed network.

10000

02000

4000

6000

8000

10000

time

(d) Conventional network.

Figure 7: Changing ε of the sigmoid function.

The simulated result is shown in Fig. 8. From thisfigure, the SA method and the proposed network withSA do not escape from local minima as well as theconventional network. We consider that the networkneeds some irregular change of neurons themselves inearly learning time to find a good solution.

[10] Y. Uwate, Y. Nishio, T. Ueta, T. Kawabe and T. Ikeguchi,

218“Performance of Chaos and Burst Noise Injected to theHopfield NN for Quadratic Assignment Problems,” IEICE
Transactions on Fundamentals, vol. E87-A, no. 4, pp. 937943, Apr. 2004.