2006 RISP International Workshop
on Nonlinear Circuits and Signal Processing (NCSP'06)
Waikiki Beach Marriott, Honolulu,Hawaii, USA, March 3-5, 2006

Performance of Feedforward Neural Network with External Influence Function
for Back Propagation Learning
Yoko Uwate † , Thomas Ott ‡ , Yoshifumi Nishio † and Ruedi Stoop ‡† Dept. of Electrical and Electronics Engineering, Tokushima University
2-1 Minami-Josanjima, Tokushima, Japan
Email: {uwate, nishio}@ee.tokushima-u.ac.jp‡Institute of Neuroinfomatics, University / ETH Zurich
Winterthurerstrasse 190, 8057 Zurich, Switzerland
Email: {tott, ruedi}@ini.phys.ethz.ch

1. Introduction
The human brain contains at least 100 billion neurons, eachwith the ability to influence many other cells. Clearly, highlysophisticated and efficient mechanisms are needed to enablecommunication among this astronomical number of elements.
Such communication is made possible by synapses, the functional contacts between neurons. The synapses have a majorpart to play for learning, memory and control in the functional
brain. The connection strengths of the synapses correspondto weights of neurons in artificial neural networks.
Back Propagation (BP) learning [1] is one of engineering
applications of artificial neural networks. The BP learning
operates with a feedforward neural network which is composed of an input layer, a hidden layer and an output layer,
and the effectiveness of the BP learning has been confirmedin pattern recognition, system control, signal processing, andso on [2]-[4]. It is an optimization produce based on gradient descent that adjusts weights to reduce the system error or
cost function. The errors are used as inputs to feedback connections from which adjustments are made to the synaptic
weights layer by layer in a backward direction.
In this study, we investigate the performance of the BP
learning if the action of synaptic weights changing by external influence. We introduce periodically sine wave as external
influence function to adjustment of the weights. By computer
simulation, the proposed network gains the good performancefor learning efficiency. Furthermore, the characteristics of theneurons in hidden layer are investigated.

gorithm. The difference lies in the timing of the update of
the weight. The update of the standard BP is performed after
each single input data, while the update of the batch BP isperformed after all different input data. The total error E of
the network is defined as the following equation.
 NPP12
Ep =(tpi − opi ) )
(1)
E=
2 i=1
p=1
p=1
where P is the number of the input data, N is the number ofthe neurons in the output layer, t pi denotes the value of thedesired target data for the pth input data, and o pi denotes thevalue of the output data for the pth input data. The goal of thelearning is to set weights between all layers of the network
to minimize the total error E. In order to minimize E, theweights are adjusted according to the following equation:
k−1,kk−1,kwi,j(m + 1) = wi,j(m) +k−1,k∆p wi,j(m) = −ηThe standard BP learning algorithm was introduced in [ 1].
The BP is the most common learning algorithm for feedforward neural networks. In this study, we use the batch BP
learning algorithm. The batch BP learning algorithm is expressed by similar formula of the standard BP learning al-p=1k−1,k∆p wi,j(m)(2)

k−1,k∂wi,jk−1,kwhere wi,jis the weight between the ith neuron of thelayer k − 1 and the jth neuron of the layer k, m is the learning time, and η is a proportionality factor known as the learning rate. In this study, we introduce inertia term in the 2nd
term of the right-hand side of Eq. (4) to improve the rate ofconvergence.
k−1,k(m) = −η∆p wi,j2. Back Propagation Algorithm∂EpP∂Epk−1,k∂wi,jk−1,k+ ζ∆p wi,j(m − 1) (3)where ζ denotes the inertia rate. This can be accomplished byadding a fraction of the previous weight change to the current
weight change. The addition of such a term can help smoothout the descent path by preventing extreme changes in the- 17 -

gradient due to local anomalies. It can act as an average effect which smooths the trajectory of the gradient as it moves
downhill.

Input

Outputxx2

3. External Influence Function
We investigate the performance of BP learning when theadjustment of the weights changes periodically by externalinfluence. The adjustment of weights equation with external
influence function is given by the following equation.

(a) Network structure.
1

k−1,k∆p wi,j(m)∂Epk−1,k= fE (m) × −η ∂wk−1,k + ζ∆p wi,j (m − 1)

0.8

(4)

0.6

x2i,j0.4

where fE is the external influence function as shown in Fig. 1.
In this study, we consider that the external influence function
is sine wave. From this figure, A 1 and A2 denote the amplitude of this function, and T is the period of the sine wave. Weset T = 1000.

0.2
0-1-0.500.5

1x(b) Learning example.
Figure 2: Network structure and learning example.

fEA2target. We define “Average Error E ave ” by the following
equation as mean square error.

IterationtimeP

Eave =A11 P p=1

1(tp − op )2
2


(5)

T

Figure 1: External influence function.

4. Simulated Results
We consider the feedforward neural network producing
outputs x2 for inputs data x as one learning example. Thesampling range of the input data is [−1.0, 1.0] and the stepsize of the input data is set to be 0.01. We carried out theBP learning by using the following parameters. The learning
rate and the inertia rate are fixed as η = 0.1 and ζ = 0.02,respectively. The initial values of the weights are given between −1.0 and 1.0 at random. The learning time is set to50000, and the 8 neurons are prepared in the hidden layer.
The network structure using this study and learning example
are shown in Fig. 2.

Figure 3 shows one example of the simulation results when
the amplitude of the external influence functions are changed.
The horizontal axis is iteration time and the vertical axis is
Eave . From this figure, we can confirm that the proposed
network with the external influence function gains better performance than the conventional network. The learning curve
converges more quickly when the external influence has large
amplitude. Eave of the proposed network and the conventional network are summarized in Tab. 1. The results showthat the proposed network gains a good performance when
the amplitude of the external influence function is large.

4.1. Performance of learning process

Table 1: Eave of the after BP learning process.
Proposed

Conventional

We investigate the convergence speed and the learning efficiency as the total error between the output and the desired- 18 -

A1 , A2
0.1, 2.0
0.1, 5.0
0.1, 10.0
0.1, 20.0

Eave
0.026374
0.000987
0.000826
0.000750
0.028067

1
Proposed network (A1, A2)=(0.1, 2.0)
Proposed network (A1, A2)=(0.1, 5.0)
Proposed network (A1, A2)=(0.1, 10.0)
Proposed network (A1, A2)=(0.1, 20.0)
Conventional network

Satisfaction

4200.01StressEave

0.1-20.001-412345678787878

Position of neuron in hidden layer

0.0001

05000

10000

15000

(a) A1 =0.1, A2 =2.0.

20000

time
4Satisfaction

Figure 3: One example of learning process.

4.2. Stress and satisfaction

2-2-4
123

4

56

Position of neuron in hidden layer(b) A1 =0.1, A2 =5.0.
4
Satisfaction

Next, the adjustments of the weights of the proposed network and the conventional network are compared. We definethat the network feel stressed if the adjustment of the weightsof the proposed network is smaller than the conventional network. On the other hand, the network feel satisfied if theadjustment of the weights of the proposed network is larger
than the conventional network. The definition of the stressand the satisfaction are shown in Fig. 4.

Stress

02

Stress

0-2-4
12

3

4

5

6

Position of neuron in hidden layer(c) A1 =0.1, A2 =10.0.

Satisfaction

420The simulated results compared the area ratio of the stressand the satisfaction are shown in Fig. 5. In this simulation,
we use the same initial conditions of the weights. We confirm that the area of the stress of the satisfaction is small
when the amplitude of the external influence function is small(Fig. 5 (a)). And the some neuron’s area of the stress or thesatisfaction becomes large by increasing with amplitude ofthe external influence function (Figs. 5 (b), (c) and (d)).

- 19 -

StressFigure 4: Stress and satisfaction of the proposed network.
-2

-4
1

2

3

4

5

6

Position of neuron in hidden layer(d) A1 =0.1, A2 =20.0.
Figure 5: Simulated results of stress and satisfaction.

0.01

0.01

Proposed network
Conventional network

0-0.005

0-0.005

-0.01
5000

10000

15000

20000

-0.01
05000

10000

time

0.01

Proposed network
Conventional network-0.005

10000

0.01

Proposed network
Conventional network-0.01

01500020000

Proposed network
Conventional network

0-0.01
05000

10000

time

15000

20000

05000

10000

time

12(d) ∆w11
.

0.01

0.01

20000

12(f) ∆w13.

Proposed network
Conventional network

0.005

∆w18

0.005

15000

time

12(e) ∆w12

Proposed network
Conventional network

20000-0.005

-0.01
10000

15000

0.005

-0.0055000

500012(c) ∆w13.

∆w16

0

00

time

0.005

∆w15

∆w14

20000

12(b) ∆w120.005

∆w17

15000

time

12(a) ∆w11.

0.01

0-0.005-0.01
0Proposed network
Conventional network

0.005∆w130.005

∆w12

∆w11

0.005

0.01

Proposed network
Conventional network

0-0.005

0-0.005

-0.01

-0.01
05000

10000

15000

20000

05000

10000

time

12(g) ∆w11.

15000

20000

time

12(h) ∆w12.

Figure 6: Adjustment of weights (A 1 =0.1, A2 =20.0).
One example of varying adjustments of the weights between the input and the output layers is shown in Fig. 6. The
adjustments of some weights of the proposed network change
widely when the iteration time is around 2000.

References[1] D.E. Rumelhart, G.E. Hinton and R.J. Williams, “Learning
Internal Representations by Error Propagation,” Parallel Distributed Processing, vol. 1, MIT Press, MA, pp. 318-362, 1986.
[2] Y. LeCun, B. Boser, J.S. Denker, et al, “Handwritten Digit
Recognition with a Back-Propagation Network,” Advances in
Neural Information Processing Systems, vol. 2, MIT Press, MA,
1990.

5. Conclusions
In this study, we investigated the performance of the BP
learning if the action of synaptic weights changing by external influence. We introduced the periodically sine wave as external influence function to the adjustment of the weights. By
computer simulation, the proposed network gains the good
performance for learning efficiency. Furthermore, the characteristics of the neurons in the hidden layer were investigated.

[3] M. Kawano, Y. Uno, M. Isobe and R. Suzuki, “A Hierarchical
Neural Network Model for Voluntary Movement with Application to Robotics,” IEEE Control Systems Magazine, vol. 8, pp.
8-16, 1988.
[4] T.J. Sejnowski and C.R. Rosenberg, “Parallel Networks thatLearn to Pronounce English Text,” Complex Systems, vol. 1, pp.
145-168, 1987.

- 20 -