2017 RISP International Workshop on Nonlinear Circuits,Communications and Signal Processing (NCSP'17)
Guam, USA, February 28th to March 3rd, 2017

Denoising Auto Encoder with Logistic Map
Ryuta YOSHIMURA, Shinsaburo KITTAKA, Yoko UWATE, and Yoshifumi NISHIO†Dept. of Electrical and Electronic Engineering, Tokushima University
2-1 Minami-Josanjima, Tokushima 770–8506, Japan
Email: {yoshimura,kittaka,uwate,nishio}@ee.tokushima-u.ac.jp

Abstract

2. Auto Encoder

Auto encoder is used as prior learning. It is unknown thatIn this study, we investigate denoising auto encoder as priorlearning. We show results using chaotic noises with denoising each layer should do, what kind of feature extraction at theauto encoder. We investigate value and parameter of network time of prior learning. Therefore, the auto encoder is unsupervised learning. An initial value can be said that it is suitable if
by using a denoising auto encoder.
an output from hidden layer can restore input. In other words,
1. Introduction
we hope the network does not lose information in the hiddenlayer. The structure of the auto encoder has three layers in
Recentry, the method of machine learning is Deep Learn- neural network consisting of an input layer, hidden layer anding. It has already used as commercial services. The compa- output layer. How to obtain a good value is minimizing thenies which have advanced technology about Deep Learning error function. By making the unit of the hidden layer smaller
have used it as image search. The reason why Deep Learning than the unit of the input layer, the dimensions of the input inbecomes hot topic is that A team who uses the new machine formation are reduced in Fig. 1. Then, when hidden layer islearning method of the Deep Learning won the championship taken out, an important fetures remain.
in competition of the image recognition in 2012.
The reason why the team using Deep Learning won is that
Deep Learning has a multi neural network. A neural networkwith two or more hidden layers is called a deep neural network. Neural network looks like human brain cells. It is
expressive because Deep Learning has very complicated relationships between their inputs and outputs if it has more andmore hidden layers. However, When there are two or more
hidden layers, it usually does not work because it takes moretime to have the whole network to learn by error back propaFigure 1: Auto encoder.
gation and overﬁtting also becomes more easy. If it is difﬁcult
to learn the whole by error back propagation, we should do 3. Denoising Auto Encoder
prior learning each layer and give initial values to make feaIn addition to the above, it is possible to obtain a better initure extraction [1].
tial
value by setting constraint conditions on the auto encoder.
The idea of auto encoder is an effectively prior learningThe
constraint conditions is to add a noise to each input. Apwith each layer of the multilayer neural network. When weplying
noise increases the number of samples and there isstack two or more hidden layers, after auto encoder ﬁrst we
nodangerof relying on certain features. Also, the features
make it with only one hidden layer, then remove the output
extracted
based
on many samples are important parameters
layer, consider the hidden layer as the input layer and stackthroughout
thenetwork.
In other words, it is good feature exone more layer in order to make deep neural network. We
traction
toworktheautoencoder under strict conditions. It isbuild the network for the purpose of using it as input of othercalleddenoisingauto
encoder.
We obtain an initial value withmodels as a certain potentiality vector by reducing the dimenthe
highergeneralization
abilityby performing prior learning
sion of the hidden layer. Furthermore, When we use inputwiththem.
Denoisingauto
encoder
is a model that removes
data with noise insted of original data, we obtain good initialnoisefromdatabylearning
anetworkthat uses the originalvalues. This method is called denoising auto encoder.
dataasacorrectanswer.
In this study, we show results using chaotic noises withdenoising auto encoder. The noises are made up from thelogistic map.

- 57 -

4. Proposed Method
Denoising auto encoder uses input data with the noise andassume original data as a correct answer. Furthermore, itlearns the model which removes a noise from the data. We
have denoising auto encoder to be able to reconstructed theorigin data reproducing noise by adjusting a parameter. We
will show an update equation of parameter.
y = s(W x + b)

(1)

z = s(W  y + b )(2)Equation (1) shows encode. Equation (2) shows decode. The
process of converting x to y is called encode, and the processof converting y to z is called decode. z means reconstructing data from x. We update weight so that output z nears
input x. s means sigmoid function. W and W  mean weight.
In this study, we set a restriction that Wx equel Wx . It iscalled tied weight and reduse eight to calculate. How closethe z to x is deﬁned by an error function. It is the learning of
the denoising auto encoder to update the parameter W , b, b
by the error back propagation in order to minimize this error
function. The cross entropy error function which is shown byequation (3) is used as error function.
LH (x, z) = −x log z − (1 − x) log(1 − z)

Figure 2: Logistic map at the time we decide the parameter aas 3.828327.
Figure 2 shows a behavior of intermittency chaos. The
part where the irregular behavior is occurring and where theperiodic behavior occurs alternately are called intermittencychaos. When we determine the parameter a as 3.828327,
the logistic map shows intermittency chaos. We generate a
random number by the logistic map and compare it with thethreshold that we set. When it does not exceed the threshold,it output 1. We mutilate it to each pixel of the input data.
The pixel which is multiplied 0 is painted black and becomea noise.

(3) 5. Simulation results

We forcus on the following two points. They are differIn order to minimize the error function, we should calculate ence due to learning loops and noise application rate. We use
a slope of LH for W , b and b . After calculating slopes of W , MNIST data set. MNSIT has 28 x 28 pixels image. Our netb and b , we obtain slopes which are shown by equation (4).
work has input layer 784 units, hidden layer 500 units andoutput layer 784 units. First, we explain simulation result dif⎧N⎪∂  ∂L
ference due to learning loops. The noise application rate is 60⎪newold⎪W=W +⎪⎪N∂W%. We show cost which is the output from the error function
⎪⎪
n=1⎪
⎪
in table 1 at any learning loops.
⎪N⎨∂  ∂L
newold=b +b(4)N n=1 ∂b⎪
⎪
⎪
⎪
⎪
N⎪
⎪
⎪ bold + ∂  ∂L⎪⎪⎩
N n=1 ∂b
When input data of the N unit are given, update of theparameter is given by stochastic gradient descent in denoising
auto encoder. Equation (5) shows the update of the parameter.
means a learning rate. The proposed system uses a logistic
map for chaos function to decide the probability.
Figure 3: A cost at learning loops.
xn+1 = axn (1 − xn )(5)

xn means input. xn+1 means output. Parameter a makes thelogistic map constant value, periodic vibration or aperiodic
complicated behavior that is called the chaos. The change
in the value of xn with to increase the n. We determine theparameter a as 3.828327.

Figure 3 shows that the cost decreases and denoising auto
encoder can learn it. The error converges after the 6th time.
We will show a visuals of the weight and reconstructed images.

- 58 -

(a)(b)

Figure 6: A cost at rate of noise application.
Figure 6 shows that cost increases when rate increases. Wewill show some images with noise. Increasing the noise rateincreases the cost. This is because it is an error between thenoise images and the original images.
(c)Figure 4: Reconstructed image of studying difference due tolearning loops. (a) Learning loops is 1. (b) Learning loops is
5. (c) Learning loops is 20.
Figure 4 shows reconstructed images. As learning loops increase we can see that it is clearly reconstructed. It is thoughtthat a tendency to correct the black place noised to white becomes strong. When the learning loops increase, this tendency becomes remarkable. We can see this tendency too
much at the learning loops of 20.

(a)

(a)

(b)

(c)

(d)

(e)

(f)

(b)

Figure 5: Weight visualization of studying difference due tolearning loops. (a) Learning loops is 1. (b) Learning loops is5.
Figure 5 shows that the tendency of weight is different withlearning loops. We thought each weight gains features byincreasing the learning loops. Each weight visualizes clearer
at the leaning loops of 20 than the learning loops of 1.
Next, we explain difference due to noise application rate.
The parameter a of the logistic map is 3.828327. Learning
loops is 10. Noise application rate is different. We show costwith some rate which is the output from error function in Fig.
6.

Figure 7: Corrupted image of studying difference due to noise
application rate. (a) 0%. (b) 20%. (c) 40%. (d) 60%. (e) 80%.
(f) 100%.
Figure 7 shows corrupted images due to noise application
rate. Corrupted image at rate of 0% means input data for auto
encoder.

- 59 -

(a)

(b)

(a)

(b)

(c)

(d)

(c)

(d)

(e)

(f)

(e)

(f)

Figure 8: Reconstructed image of studying difference due to Figure 9: Weight visualization of studying difference due tonoise application rate. (a) 0%. (b) 20%. (c) 40%. (d) 60%. noise application rate. (a) 0%. (b) 20%. (c) 40%. (d) 60%.
(e) 80%. (f) 100%.
(e) 80%. (f) 100%.
Figure 8 shows reconstructed images. The more the rate ofnoise is increased, the stronger the tendency to correct noise
Acknowledgment
to white. Therefore, the letters are visually thick.
This work was partly supported by JSPS Grant-in-Aid for
Scientiﬁc Research 16K06357.
Figure 9 shows tendency of weight is different due to noise
application rate. When it is 80% it seems the weight has theReferences
most characteristic amount like letter of number.
[1] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky,
Ilya Sutskever and Ruslan Salakhutdinov,“Dropout: ASimple Way to Prevent Neural Networks from Overﬁtting,”Journal of Machine Learning, pp.1929-1958, Jun.
2014.

6. Conclusion
In this study, we investigated denoising auto encoder asprior learning. We used some noises made up from the logistic map. We used intermittency chaos are made from logistic
map by adjusting parameter. We investigated cost, weightand reconstructed images. When learning loops increased,
the images were reconstructed clearly. It was thought that
this is because a tendency to correct black noise to white becomes strong. Visualized weights showed features that varied in each when learning loops increased. The letters of thereconstructed images were bigger than an original and werereconstructed clearly when the rate of noise increased. Someof the visualized weights showed features like number.

- 60 -