2017 RISP International Workshop on Nonlinear Circuits,Communications and Signal Processing (NCSP'17)
Guam, USA, February 28th to March 3rd, 2017

Improvement of Learning Efﬁciency Using Noise for Back Propagation in Deep Learning
Kazuki NAGAO, Shinsaburo KITTAKA, Yoko UWATE and Yoshifumi NISHIO
Department of Electrical and Electronic Engineering of Tokushima University
2-1 Minami-Josanjima, Tokushima, Tokushima 770-8506, Japan
E-mail: {k-nagao, kittaka, uwate, nishio}@ee.tokushima-u.ac.jp

Abstract
Deep learning is made to imitate the human nervous system. Since 1980s, there have been concepts and methods ofdeep learning. Currently, deep learning is becoming the third
artiﬁcial intelligence boom. In this study, we propose a system of deep learning. This system is noisy back propagation.
Weights of neurons are optimized using a noise in the backpropagation. The purposes of this study are to improve learning accuracy and to reduce learning loops. We use two data Figure 1: Recognition of handwritten numbers by deep learnset. We compare the proposed system with the conventional ing.
system.
1. Introduction
Human nervous system has neurons which are connected.
Then, human can think things. Deep learning is made to imitate the human nervous system. Since 1980s, there have been
concepts and methods of deep learning. Deep learning waspaid a lot of attention in the ﬁeld of image recognition from2010. Currently, deep learning is becoming the third artiﬁcial
intelligence boom.
Deep learning is a superposition of neural networks and has
deep structure. Deep learning has more four layers include
input layer and output layer. It is repeated learning by thetime and the input data are gradually being transferred to thedeeper and deeper from the ﬁrst layer. Previously, researchersand engineers had set the parameters manually. Now, thatdeep learning automatically learns valid features of discrimination. It has been improved accuracy of pattern recognition.
It is attracting attention.
In June 2012, it was able to recognize the cat on computer
by research on Google. After that, it comes to be used in various ﬁelds. For example, image ﬁeld, emotional awareness,
module of self-driving, medical ﬁeld, ﬁnancial ﬁeld and so on[1]. Google released a software library of machine learningtechniques including deep learning in 2015. We raised interest in deep learning. Figure 1 shows recognition of handwritten numbers by deep learning with Google’s software library[2]. We can see that number are learned from the left in order.
It takes the character of the number and judges which number
it is.

In deep learning, there are a variety of techniques in orderto improve the learning accuracy. Technique of back propagation compares input data and output data to ﬁnd deviations offeature values and optimize weights [3]. This technique canreduce learning loops. If the learning period is too long or
the training data is not typical, it adapts to features unrelated
to the features that should be originally learned. This phenomenon is called over learning. In such a case, it uses themethod of dropout to correspond to input data. Deep learning
requires a large amount of learning data. When there are notenough input data, input data are shifted slightly due to use
noise to increase features. There are various techniques likethese.
In this study, we propose a system of deep learning. Thissystem is noisy back propagation. Weights of neurons are optimized using a noise in the back propagation. The purposesof this study are to improve learning accuracy and to reduce
learning loops. We compare proposed system with the conventional system using two data sets.
2. Proposed method
Figures 2 and 4 show the conventional systems. Figures 3and 5 show the proposed systems. Arrows on left to right(h1 , h2 ) indicate propagation of input signal and right to left
(h1back , h2back , yback ) indicate learning of back propagation.
x, y and w show input data, output data and weight between
neurons.
(A) Iris data set [4]The iris data set is data to classify into 3 types of iris with- 397 -4 features. There is a total of 150 data set. We use a model
consisting of two layers of intermediate layers and Gaussian
distribution (0, 0.01) for noise and add it to equations (2). Inaddition, the putting place of noise is changed. We experiment 8 patterns for the conventional system without noise,(h1back ), (h2back ), (yback ), (h1back h2back ), (h1back yback ),(h2back yback ) and (h1back h2back yback ). These combinations indicate where noise is added. We deﬁne the learning
loops = 25000 and the number of learning data sets = 150.

value of the weights is chosen by random from 0.04 or less.
(B) Cars data set [4]The cars data set is ﬁve items of car price and safety asinput, and outputs as four evaluations. There is a total of
1728 data set. We use a model consisting of three layersof intermediate layers and Gaussian distribution (0, 0.01) fornoise and add it to equations (5). In addition, the putting
place of noise is changed. We experiment 15 patterns forthe conventional system without noise, (h1back ), (h2back ),(h3back ), (yback ), (h1back h2back ), (h1back h3back ), (h1backyback ), (h1back h2back h3back ), (h1back h2back yback ), (h1backh3back yback ), (h1back h2back h3back yback ), (h2back h3back ),(h2back yback ), (h2back h3back yback ) and (h3back yback ).
These combinations indicate where noise is added. We deﬁne as the learning loops = 1000 and the number of learning
data sets = 1728.

Figure 2: A schematic drawing of deep learning consisting oftwo intermediate layers.

Figure 4: A schematic drawing of deep learning consisting ofthree intermediate layers.

Figure 3: A schematic drawing of proposed deep learning
consisting of two intermediate layers.
⎧1
⎪
⎪

h1 =
⎪
⎪
(−(w1 x))
⎪1+e
⎪
⎨
1

h2 =
⎪
1 + e(− (w2 h1 ))
⎪⎪
⎪
1⎪
⎪
⎩ x=
(−(w3 h2 ))1+e(1)⎧− ty)(1 − y)y⎪⎨ yback = (yh2back =w y
(1 − h2 )h2 3 back⎪⎩ h=w h(1 − h )h1back

2 2back⎧(l+1)⎪= w1l − εxh1back⎨ w1(l+1)
w2= w2l − εh1 h2back⎪⎩ (l+1)w3= w3l − εh2 yback

1

Figure 5: A schematic drawing of proposed deep learning
consisting of three intermediate layers.
(2)
1(3)

ε is the rate of decay, which is 0.01. h means activation
function. w means weights between neurons. The initial- 398 -⎧1⎪⎪⎪ h1 =⎪(−(w1 x))⎪1+e⎪⎪⎪1⎪⎪⎨ h2 =1 + e(− (w2 h1 ))1⎪⎪
⎪ h3 =⎪⎪
1 + e(− (w3 h2 ))
⎪⎪
⎪
1⎪⎪

⎩ y =
1 + e(− (w4 h3 ))(4)⎧yback = (y− ty)(1 − y)y⎪⎪⎪⎪
⎨ h3back =w4 yback (1 − h3 )h3h=w3 h3back (1 − h2 )h2⎪
2back⎪⎪⎪⎩ h
w2 h2back (1 − h1 )h1
1back =
⎧w1 (l+1) = w1 l − εxh1back⎪⎪⎨w2 (l+1) = w2 l − εh1 h2back⎪w (l+1) = w3 l − εh2 h3back⎪⎩ 3 (l+1)
w4= w4 l − εh3 yback

Table 2: Learning loops of model (A).
h1backh2backybackh1back h2back
h1back yback

(5)

(6)

ε is the rate of decay, which is 0.4. h means activation function. w means weights between neurons. The initial value ofthe weights is chosen by random from 0.3 or less.
3. Simulation result(A) Iris data set
Figure 6 shows the change of the learning accuracy andlearning loops of a conventional system and proposed system. Table 1 shows minimum learning accuracy for all kindsof patterns. Table 2 shows learning loops when learning accuracy is 0.1 or less.
From Figure 6, the learning accuracy and learning loops byusing Gaussian distribution (0, 0.01) are better than not using
it. From Table 1, most results by using the noise are found tobe better.

loops
16878
24668
24984
22974
15599

(B) Cars data setFigure 7 shows the learning accuracy and learning loops ofa conventional system and proposed systems. Table 3 showslearning accuracy of minimum value of all kinds of patterns.
Table 4 shows learning loops for all kinds of patterns whenlearning accuracy is 0.064 or less.
From Figure 7, the learning accuracy and learning loops byusing Gaussian distribution (0, 0.01) are better than not usingit.

Figure 7: The simulation results of a schematic drawing ofdeep learning consisting of three intermediate layers.

Table 3: Learning accuracy of model (B).

Figure 6: The simulation results of a schematic drawing ofdeep learning consisting of two intermediate layers.
Table 1: Learning accuracy of model (A).
Conventional
h1backh2backybackh1back h2back
h1back ybackh2back ybackh1back h2back ybackaccuracy
0.155788
0.074552
0.094836
0.099584
0.076192
0.025454
0.119983
0.136539- 399 -

Conventional
h1backh2backh3backybackh1back h2back
h1back h3backh2back h3back
h1back h2back h3back
h1back ybackh2back ybackh3back ybackh1back h2back ybackh1back h3back ybackh2back h3back ybackh1back h2back h3back yback

minimum
0.063878
0.064484
0.06359
0.067046
0.059693
0.063429
0.066899
0.06809
0.067258
0.0604970.06114
0.066832
0.080702
0.064958
0.072979
0.082566

Table 4: Learning loops of model (B).
ybackh1back ybackh2back ybackloops
427338410

4. Conclusion
In this study, we proposed technique of deep learning. It isusing noise at back propagation. In the proposed technique,we choose Gaussian distribution (0, 0.01) for noise. Then,we examined whether this technique is effective in simulationthat is using two data sets.
In simulations, we were able to improve learning accuracy
and reduce learning loops to use noise at back propagation.
From the tendency of research results of two data sets, wefound that results are improved by using noise at back propagation from the output layer to the ﬁnal intermediate layer.
Because back propagation of the output layer is comparedwith input data and output data, so it is properly carried out
weight adjustment.
In the future work, we will use other data sets to demonstrate usability and a variety of noise. Then, we obtain thebest performance of the proposed technique.
Acknowledgment
This work was partly supported by JSPS Grant-in-Aid for
Scientiﬁc Research 16K06357.
References[1] Y. Iwatani and H. Furusho, “Lane Recognition by Deep
Learning for Autonomous Driving on City Roads,” Proceedings of Nissan technical review, pp. 40 - 46, 2016[2] Y. Le Cun, B. Boser, J. S. Denker, D. Henderson, R.
E. Howard, W. Hubbard and L. D. Jackel, “Handwritten
Digit Recognition with a Back-Propagation Network,”
Proceedings of Advances in Neural Information Processing Systems, pp. 396 - 404, 1990
[3] Rumelhart, E. David, E. G. Hinton and J. R. Williams,“Learning representations by back-propagating errors,”
Cognitive modeling 5.3, 1988
[4] A.
Asuncion
andD.
Newman(2007),“Machine
Learning
Repository,”
https://archive.ics.uci.edu/ml/about.html- 400 -