2004 RISP International Workshop on Nonlinear
Circuit and Signal Processing (NCSP'04)
Hawaii, USA, Mar. 5-7, 2004.

An Improvement of Learning Efficiency by Noise Adding
to a Feed-Forward Neural Network
Koki Nishizono , Yoko Uwate and Yoshifumi Nishio
Tokushima University
2-1 Minami-Josanjima, Tokushima 770-8506, Japan
Tel: +81-88-656-7470
Fax: +81-88-656-7471
Email:  koki, uwate, nishio @ee.tokushima-u.ac.jp

1. Introduction

0.5
0.4

As a learning algorithm of feed-forward neural networks,
the error reverse propagation learning (BP, Back Propagation)
method is used widely. However, the BP method requires alarge amount of time because it needs repeated learning.
In this study, in order to improve learning efficiency ofa feed-forward neural network, a technique adding various
sizes and patterns of noise to the connection weights between
the input layer and the middle layer or between the middle
layer and the output layer is proposed. At first, a noise withvarious sizes is added to learn exclusive logical sum (EXOR),
which is an example of easy problems. Next, a noise divided
to various patterns is added to learn some grayscale patterns,
which is a somewhat complicated problem.

0.3
0.2
0.1
0-0.1
-0.2
-0.3
-0.4
-0.5
0500

1000

1500
time

2000

2500

3000

Figure 1: Uniform noise.
2. Noise to be Added
The standard noise used in this research is a uniform noise
spread over an interval [-0.5, 0.5] as shown in Fig. 1. This
noise is added to the connection weights between the input
layer and the middle layer or between the middle layer and theoutput layer of a feed-forward neural networks. As varying
the amplitude of the noise by multiplying an integer between
0 and 8, the learning time of the networks is investigated.
3. EXOR Problem
Figure 2 shows the feed-forward neural network used for
the EXOR problem. Number of the middle layer is one. Accordingly, this network consists of 3 layers. The input layer
is composed of two elements and the output layer is composed of one. The number of the middle layer’s elements
 is changed from 3 to 15. A teacher signal is given to
the input layer and the weights are updated according to theBP method. The learning is repeated until an error signal becomes smaller than a certain value (in this research, this value
is set to ).

Figure 3 shows some of the simulation results for theEXOR problem. The figure shows the relation between theamplitude of the uniform noise and the average of 100 simulation results of the learning time.
  isWhen the number of the middle layer’s elements
small, the learning time is almost constant regardless of the  is large, the learning
amplitude of the noise. While whentime varies depending on the amplitude of the noise and itis not good for a wide range of the amplitude of the noise.
But, we should mention that the best learning time is obtained   .
from the case of
4. Pattern Recognition
Next, we consider a pattern recognition problem. Figure 4
shows four gray scale patterns used for the pattern recognition
problem. Figure 5 shows the feed-forward neural network
used for this problem. The input and the middle layers havefour elements and the output layer has two elements.
At first, the uniform noise with various sizes of the ampli-- 133 -Input layer

Middle layerOutput layer
noise

000

1

101

1

Figure 2: Feed-forward neural network for EXOR problem.
Figure 4: Four grayscale patterns.

2000

1800

Input layer
11600

Middle layer

Output layer

alearning time

1400

1200
middle layer’s element = 5
middle layer’s element = 10
middle layer’s element = 15

1000

2b800

600

3c

400

200
012

3

4
amplitude of noise

56784
dFigure 3: Simulated results for EXOR problem.

1500
1400
1300
1200

learning time

tude is added to all the connection weights between the input
layer and the middle layer.
Figure 6 shows the simulation results for the pattern recognition problem. From the figure, we can say the best result is  and the best learning time
obtained from the case ofis 527.08.
Next, we consider changing the input patterns of the noise.
Namely, we choose the weights to which the noise is added.
Because there are 16 weights between the input layer and themiddle layer, the number of the combinations of the choosing   . In order to reduce the computation
becomescost, we carry out computer simulations for the four differentcases. Namely, the first case is adding the noise to only theweights between the element 1 and the middle layer and thesecond case is adding the noise to only the weights between
the element 2 and the middle layer. Similarly, the third andfour cases are adding the noise to only the weights from theelements 3 and 4, respectively. Tables 1, 2, 3, and 4 summarize the learning times for the four cases. In the tables, “1”   is added to the correspondmeans that the noise withing weight, while “0” means that no noise is added to theweight. From Table 1, the two best learning times with noiseto   are obtained for the cases of  0,1,1,1  and  1,1,1,1  .
From Table 2, the two best learning times with noise to ! 

Figure 5: Feed-forward neural network for pattern recognition problem.

1100
1000
900
800
700
600
500
012345678

amplitude of noise

Figure 6: Simulated results for pattern recognition problem.

- 134 -

are obtained for the cases of  1,1,1,0  and  1,1,1,1  . FromTable 3, the two best learning times with noise to !"  are obtained for the cases of  0,1,1,1  and  1,1,1,1  . From Table 4,
the two best learning times with noise to   are obtained
for the cases of  0,1,1,1  and  1,1,1,1  . Hence, we consider
combining the cases. For the sixteen combinations using thetwo best cases in the four tables, we carried out computer
simulations. The results are summarized in Table 5. The bestlearning time is 438.63 and this is smaller than the learning
time 527.08 obtained when the noise is added to all of theweights.
Therefore, we can say that adding noise to the connection
weights can improve the learning time of the pattern recognition problem. However, selecting the weights to which noiseis added and the amplitude of the noise is important to obtain
good results.
5. Conclusions
In this study, in order to improve learning efficiency of afeed-forward neural network, a technique adding the various
sizes and patterns of noise to the connection weights between
the input layer and the middle layer or between the middle
layer and the output layer has been proposed.
We confirmed the effectiveness of the method for the twoexamples.
References[1] Itsuo Kumazawa, Learning and Neural Networks,
Morikita Shuppan, 1998.
[2] Norio Baba, Fumio Kojima and Seiichi Ozawa, Basesand Applications of Neural Networks, Kyoritsu Shuppan,
1994.
[3] Yoshikazu Nishikawa and Shinzou Kitamura, Neural
Networks as Applied to Measurement and Control,
Asakura Syoten, 1995.

- 135 -Table 1: Learning time with noise to  # #%  &  ' learning time
00011000.27
0010984.67
0011955.52
0100979.320101961.84
0110943.910111903.25
1000986.47
100
1961.53
1010954.15
1011933.72
1100951.47
1101931.75
1110943.52
1111912.63

 $Table 2: Learning time with noise to   %  &  ' learning time000
1946.120010930.5300
1
1866.780100931.400101868.04
0110868.08
0111810.42
1000942.03
1001886.41
1010871.04
1011811.55
1100866.78
1101805.79
1110796.65
111
1767.40 $Table 3: Learning time with noise to (" )" $ )" % )" & )" ' learning time
000
1872.640010863.8100
11749.740
100867.160101755.380
110757.110111657.65
1000878.35
1001773.94
1010773.04
10
11675.941100766.43
1
101660.42
1110683.44
1111600.28

Table 4: Learning time with noise to   $  %  &  ' learning time
0001871.05
00
10862.72
0011761.00
0100861.33
0101774.26
0110753.800111633.50
1000876.25
100
1763.611010773.82
1011668.591100754.12
1
101656.061
110638.631
1
1
1562.56
Table 5: Learning time with noise to all weight connections

 $ %

 #& ') $) %) &) ')" $)" %)" &)" '

 $

 %

 & '

0000000011111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111110000111
10000111001
1001
1001
10011111111111111111111111111111111111111111111110
10
10101010101011111111111111111111111111111111111111111111
1- 136 -

learning time
583.52539.26542.52563.61
589.46550.21
538.69438.63595.90566.18578.53501.95590.74538.48546.92