IEEE Workshop on Nonlinear Circuit Networks
December 9-10, 2016

Deep Learning Improving Learning Efficiency byUsing Noise for Back Propagation
Kazuki NAGAO, Shinsaburo KITTAKA, Yoko UWATE and Yoshifumi NISHIO
Dept. of Electrical and Electronic Engineering of Tokushima University
2-1 Minami-Josanjima, Tokushima-shi, Tokushima-ken 770-8506, Japan
Email: {k-nagao, kittaka, uwate, nishio}@ee.tokushima-u.ac.jp

Abstract—Deep learning began to be rapidly attention suchas image recognition from the 2010. Now, the third time ofartificial intelligence is boom. Deep learning is made to imitate thehuman nervous system. It automatically learns valid features ofdiscrimination. That is why it is popular. In this study, we study
about deep learning using noise for back propagation. Then, we
improve the learning accuracy and reduce learning loops.

I. I NTRODUCTION
Human nervous system has neurons which are connected. Itis controlled by the strength of neurons. Then, human can think
things. Deep learning is made to imitate the human nervous
system[1]. There are concepts and methods of deep learningfrom before and after 1980. It began to be rapidly attention
such as image recognition from the 2010. Now, the third timeof artificial intelligence is boom.
Deep learning is superposition of neural networks and has
deep structure. Deep learning has more three layers include
input layer and output layer. It is repeated learning by thetime, and the input data is gradually being transferred to thedeeper and deeper from the first layer. Previously, researchersand engineers had set the parameters manually. Now, that deeplearning automatically learns valid features of discrimination.
It has been improved accuracy of pattern recognition. It isattracting attention.

various fields. For example, image field, emotional awareness,
module of self-driving, medical field, financial field and soon... Google released a software library of machine learningtechniques including deep learning in 2015, and we raisedinterest in deep learning. Figure 1 shows recognition ofnumbers by deep learning with Google’s software library[2].
In deep learning, there is a variety of techniques in order toimprove the learning accuracy. Technique of back propagationcompares input data and output data to find deviations offeature values and optimize weights. This technique can reduce
learning loops. If the learning period is too long or the training
data is not typical, it adapts to features unrelated to the featuresthat should be originally learned. This phenomenon is calledover learning. In such a case, it is used the method of dropout
to correspond to input data. Deep learning requires a large
amount of learning data. When there is not too much input
data, input data is shifted slightly due to use noise to increase
features. There are various techniques like these.
In this study, we propose new system of deep learning. Thissystem is noisy feed back. Weights of neurons are optimized
using a noise in the back propagation. Purposes of this study
are to improve learning accuracy and to reduce learning loops.
There are two patterns which are a small deep learning model(A) and a deep learning model (B). We compared conventional
system and two patterns of proposed system. 1II. P ROPOSED METHOD
Figs. 2 and 4 show conventional system. Figs. 3 and 5 show
proposed system. Arrows of left to right indicate propagationof input signal and right to left indicate learning of backpropagation.

Fig. 1.

Recognition of numbers by deep learning.

In June 2012, it was able to recognize the cat on computer
by research of Google. After that, it came to be used in(A) Iris data set
We use a model consisting of two layers of intermediate
layers and Gaussian distribution (0, 0.01) for noise and add
it to equations (2). In addition, the putting place of Gaussian
distribution (0, 0.01) is changed. We experiment 8 patterns
about conventional system without noise, (h1back ), (h2back ),(yback ), (h1back , h2back ), (h1back , yback ), (h2back , yback ) and(h1back , h2back , yback ). These combinations indicate where
noise is added.

-1-1 This research is that we add one dataset to what was announced at SJCIEE.

patterns about conventional system without noise, (h1back ),(h2back ), (h3back ), (yback ), (h1back , h2back ), (h1back , h3back ),(h1back , yback ), (h1back , h2back , h3back ), (h1back , h2back ,yback ), (h1back , h3back , yback ), (h1back , h2back , h3back , yback ),(h2back , h3back ), (h2back , yback ), (h2back , h3back , yback ) and(h3back , yback ). These combinations indicate where noise isadded.

Fig. 2.
layers.

Schematic drawing of deep learning consisting of two intermediate

Fig. 4. Schematic drawing of deep learning consisting of three intermediatelayers.

Fig. 3. Schematic drawing of proposed deep learning consisting of twointermediate layers.

1∑
h1 =

(−w1 x)1+e1∑h2 =(−w2 h1 )
1+e
1
∑
 y=
1 + e(− (w3 h2 ))Fig. 5. Schematic drawing of proposed deep learning consisting of threeintermediate layers.

− ty)(1 − y)y yback = (y∑h2back =w y(1 − h2 )h2∑ 3 back h=w h(1 − h )h1back

2 2back(l+1)= w1 l − εxh1back w1(l+1)
w= w2 l − εh1 h2back 2 (l+1)
w3= w3 l − εh2 yback

1

∑
h1 =

(−w1 x)1+e1∑
 h2 =1 + e(− w2 h1 )
1

∑
h3 =


(−w3 h2 )
1+e

1
∑
 y=
1 + e(− (w4 h3 ))yback = (y∑− ty)(1 − y)y h3back =w y
(1 − h3 )h3∑ 4 backh2back =w h(1 − h2 )h2∑ 3 3back h1back =w2 h2back (1 − h1 )h1
w1 (l+1) = w1 l − εxh1back
w2 (l+1) = w2 l − εh1 h2backw (l+1) = w3 l − εh2 h3back 3 (l+1)
w4= w4 l − εh3 yback

(1)

1(2)

1(3)

ε is the rate of decay which is 0.01. h means activation
function. w means weights between neurons. The initial valueof the weights is chosen by random from 0.04 or less.
(B) Cars data set
We use Gaussian distribution (0, 0.01) for noise and add
it to equations (5). In addition, the putting place of Gaussian distribution (0, 0.01) is changed. We experiment 15(4)

(5)

(6)

ε is the rate of decay which is 0.4. h means activation
function. w means weights between neurons. The initial valueof the weights is chosen by random from 0.3 or less.

-2-III. S IMULATION RESULT(A) Iris data set
Learning accuracy and learning loops are better to besmaller. We define as the learning loops = 25000 and thenumber of learning data sets = 150.
Figure 6 shows change of the learning accuracy and learning
loops of conventional system and proposed system. Table I
shows learning accuracy for all kinds of patterns. Table IIshows learning loops when learning accuracy is 0.15 or less.
From Figure 6, the learning accuracy and learning loops ofusing Gaussian distribution (0, 0.01) are better than not using
it. From Table I, most results by using the noise are found tobe better.

From Figure 7, the learning accuracy and learning loopsof using Gaussian distribution (0, 0.01) are better than notusing it. From Table IV, h2back yback and yback are better thanconventional about minimum value and average value.

Fig. 7. Simulation results of schematic drawing of deep learning consistingof three intermediate layers.

Fig. 6. Simulation results of schematic drawing of deep learning consistingof two intermediate layers.
TABLE IL EARNING ACCURACY OF MODEL (A).
accuracy
Conventional
0.1771h1back
0.1787h2back0.1727yback
0.0269h1back h2back
0.1750h1back yback
0.0178h2back yback
0.1708h1back h2back yback
0.0261TABLE II
L EARNING LOOPS OF MODEL (A).
loopsyback12612
h1back yback8002
h1back h2back yback
11103(B) Car data set
Learning accuracy and learning loops are better to besmaller. We define as the learning loops = 1000 and thenumber of learning data sets = 1728.
Figure 7 shows the learning accuracy and learning loops
of conventional system and proposed systems. Table III showslearning accuracy of minimum value and average for all kindsof patterns. Table IV shows learning loops for all kinds ofpatterns when learning accuracy is 0.07 or less.

-3-TABLE III
L EARNING ACCURACY OF MODEL (B).
minimumaverage
Conventional
0.052
0.0634h1back
0.060
0.0734h2back
0.057
0.0697h3back
0.064
0.0639yback
0.050
0.0591h1back h2back
0.060
0.0667h1back h3back
0.058
0.0702h2back h3back
0.056
0.0723h1back h2back h3back
0.057
0.0650h1back yback
0.054
0.0660h2back yback
0.049
0.0595h3back yback
0.057
0.0718h1back h2back yback
0.052
0.0707h1back h3back yback
0.053
0.0678h2back h3back yback
0.054
0.0738h1back h2back h3back yback
0.054
0.0724

TABLE IV
L EARNING LOOPS OF MODEL (B).
loops
Conventional29
h1back11
h2back84
h3back6yback
2
h1back h2back
6
h1back h3back76
h2back h3back46
h1back h2back h3back
10
h1back yback45
h2back yback
2h3back yback39
h1back h2back yback221
h1back h3back yback8h2back h3back yback5h1back h2back h3back yback
13IV. C ONCLUSION
In this study, we proposed new technique of deep learning. It
is using noise at back propagation. In the proposed technique,we choose gaussian distribution (0, 0.01) for noise. Then, we
examined whether this technique is effective in simulation thatis using two data sets.
In simulations, we were able to improve learning accuracy
and reduce learning loops to use noise at back propagation.
From the tendency of research results of two data sets, wefound that results are improved by using noise at back propagaion from the output layer to the final intermediate layer.
Because back propagation of the output layer is comparedwith input data and output data, so it is properly carried out
weight adjustment.
In the future work, we use other data sets to demonstrateusability and a variety of noise. Then, we obtain the good
performance of the proposed technique.
ACKNOWLEDGMENT
This work was partly supported by JSPS Grant-in-Aid forChallenging Exploratory Research 26540127.
R EFERENCES[1] Chihiro Ikuta, Yoko Uwate and Yoshihumi Nishio, ”Investigation of Behavior of Deth of Neuron and Neurogenesis in Multi-Layer Perceptron,”2013 IEEE Workshop on Nonlinear Circuit Networks, pp. 65-68, 2013[2] Dumitru Erhan, Yoshua Bengio, Aaron Courville, Pierre-Antoine Manzagol, Pascal Vincent, Samy Bengio, ”Why Does Unsupervised Pretraining Help Deep Learning?,” Journal of Machine Learning Research11, pp. 625-660, 2010[3] Shohei Gotoda, Yoko Uwate and Yoshihumi Nishio, ”Cellular NeuralNetworks with Changing Templates for Image processing,” 2015 IEEE
Workshop on Nonlinear Circuit Networks, pp.21-23, 2015[4] Shinsaburo Kittaka, Ryota Oshima, Yoko Uwate and Yoshifumi Nishio,”Feed-forward Neural Netwotks with Changing Sigmoid Functions,”
Journal of Shikoku-Section Joint Convention of the Institutes of Electrical and Related Engineers, No.1-27, p.27, 2015-4-