Extracting and Composing Robust Features with Denoising
Autoencoders

Pascal Vincent
vincentp@iro.umontreal.ca
Hugo Larochelle
larocheh@iro.umontreal.ca
Yoshua Bengio
bengioy@iro.umontreal.ca
Pierre-Antoine Manzagol
manzagop@iro.umontreal.ca
Université de Montréal, Dept. IRO, CP 6128, Succ. Centre-Ville, Montral, Qubec, H3C 3J7, Canada

Abstract
Previous work has shown that the difficulties in learning deep generative or discriminative models can be overcome by an initial unsupervised learning step that maps inputs to useful intermediate representations.
We introduce and motivate a new training
principle for unsupervised learning of a representation based on the idea of making thelearned representations robust to partial corruption of the input pattern. This approach
can be used to train autoencoders, and these
denoising autoencoders can be stacked to initialize deep architectures. The algorithm canbe motivated from a manifold learning andinformation theoretic perspective or from agenerative model perspective. Comparative
experiments clearly show the surprising advantage of corrupting the input of autoencoders on a pattern classification benchmark
suite.

1. Introduction
Recent theoretical studies indicate that deep architectures (Bengio & Le Cun, 2007; Bengio, 2007) may beneeded to efficiently model complex distributions andachieve better generalization performance on challenging recognition tasks. The belief that additional levelsof functional composition will yield increased representational and modeling power is not new (McClelland et al., 1986; Hinton, 1989; Utgoff & Stracuzzi,
2002). However, in practice, learning in deep architectures has proven to be difficult. One needs only
Appearing in Proceedings of the 25 th International Conference on Machine Learning, Helsinki, Finland, 2008. Copyright 2008 by the author(s)/owner(s).

to ponder the difficult problem of inference in deep
directed graphical models, due to “explaining away”.
Also looking back at the history of multi-layer neuralnetworks, their difficult optimization (Bengio et al.,2007; Bengio, 2007) has long prevented reaping the expected benefits of going beyond one or two hidden layers. However this situation has recently changed withthe successful approach of (Hinton et al., 2006; Hinton& Salakhutdinov, 2006; Bengio et al., 2007; Ranzato
et al., 2007; Lee et al., 2008) for training Deep Belief
Networks and stacked autoencoders.
One key ingredient to this success appears to be theuse of an unsupervised training criterion to performa layer-by-layer initialization: each layer is at firsttrained to produce a higher level (hidden) representation of the observed patterns, based on the representation it receives as input from the layer below,
by optimizing a local unsupervised criterion. Eachlevel produces a representation of the input patternthat is more abstract than the previous level’s, because it is obtained by composing more operations.
This initialization yields a starting point, from which
a global fine-tuning of the model’s parameters is thenperformed using another training criterion appropriate
for the task at hand. This technique has been shown
empirically to avoid getting stuck in the kind of poor
solutions one typically reaches with random initializations. While unsupervised learning of a mapping that
produces “good” intermediate representations of the
input pattern seems to be key, little is understood regarding what constitutes “good” representations for
initializing deep architectures, or what explicit criteria may guide learning such representations. We knowof only a few algorithms that seem to work well forthis purpose: Restricted Boltzmann Machines (RBMs)
trained with contrastive divergence on one hand, andvarious types of autoencoders on the other.
The present research begins with the question of what

Extracting and Composing Robust Features with Denoising Autoencoders

explicit criteria a good intermediate representation
should satisfy. Obviously, it should at a minimum retain a certain amount of “information” about its input,
while at the same time being constrained to a given
form (e.g. a real-valued vector of a given size in thecase of an autoencoder). A supplemental criterion thathas been proposed for such models is sparsity of therepresentation (Ranzato et al., 2008; Lee et al., 2008).
Here we hypothesize and investigate an additional specific criterion: robustness to partial destructionof the input, i.e., partially destroyed inputs should
yield almost the same representation. It is motivatedby the following informal reasoning: a good representation is expected to capture stable structures in theform of dependencies and regularities characteristic ofthe (unknown) distribution of its observed input. For
high dimensional redundant input (such as images) atleast, such structures are likely to depend on evidence
gathered from a combination of many input dimensions. They should thus be recoverable from partial
observation only. A hallmark of this is our humanability to recognize partially occluded or corrupted images. Further evidence is our ability to form a highlevel concept associated to multiple modalities (suchas image and sound) and recall it even when some of
the modalities are missing.
To validate our hypothesis and assess its usefulness as
one of the guiding principles in learning deep architectures, we propose a modification to the autoencoder
framework to explicitly integrate robustness to partially destroyed inputs. Section 2 describes the algorithm in details. Section 3 discusses links with otherapproaches in the literature. Section 4 is devoted toa closer inspection of the model from different theoretical standpoints. In section 5 we verify empiricallyif the algorithm leads to a difference in performance.
Section 6 concludes the study.

2. Description of the Algorithm
2.1. Notation and Setup
Let X and Y be two random variables with joint probability density p(X, Y ), with marginal distributions
p(X) and p(Y ). Throughout the text, we will use!the following notation: Expectation: EEp(X) [f (X)] =p(x)f (x)dx.
Entropy:
IH(X) = IH(p) =EEp(X) [− log p(X)]. Conditional entropy: IH(X|Y ) =EEp(X,Y ) [− log p(X|Y )]. Kullback-Leibler divergence:
IDKL (p"q) = EEp(X) [log p(X)q(X) ]. Cross-entropy: IH(p"q) =EEp(X) [− log q(X)] = IH(p) + IDKL (p"q). Mutual information: I(X; Y ) = IH(X) − IH(X|Y ). Sigmoid: s(x) =1T
1+e−x and s(x) = (s(x1 ), . . . , s(xd )) . Bernoulli dis-tribution with mean µ: Bµ (x).
Bµ (x) = (Bµ1 (x1 ), . . . , Bµd (xd )).

and by extension

The setup we consider is the typical supervised learning setup with a training set of n (input, target) pairs
Dn = {(x(1) , t(1) ) . . . , (x(n) , t(n) )}, that we suppose
to be an i.i.d. sample from an unknown distribution
q(X, T ) with corresponding marginals q(X) and q(T ).
2.2. The Basic Autoencoder
We begin by recalling the traditional autoencoder
model such as the one used in (Bengio et al., 2007)
to build deep networks. An autoencoder takes aninput vector x ∈ [0, 1]d , and first maps it to a hid"
den representation y ∈ [0, 1]d through a deterministic
mapping y = fθ (x) = s(Wx + b), parameterized by
θ = {W, b}. W is a d! × d weight matrix and b
is a bias vector. The resulting latent representation
y is then mapped back to a “reconstructed” vectorz ∈ [0, 1]d in input space z = gθ" (y) = s(W! y + b! )with θ! = {W! , b! }. The weight matrix W! of thereverse mapping may optionally be constrained by
W! = WT , in which case the autoencoder is said tohave tied weights. Each training x(i) is thus mapped
to a corresponding y(i) and a reconstruction z(i) . The
parameters of this model are optimized to minimize
the average reconstruction error:
1 " # (i) (i) $L x ,zn i=1n

θ" , θ!" =arg min=arg min

θ,θ "

θ,θ "

n
$
1 " # (i)L x , gθ" (fθ (x(i) )) (1)n i=1where L is a loss function such as the traditionalsquared error L(x, z) = "x − z"2 . An alternative loss,
suggested by the interpretation of x and z as either
bit vectors or vectors of bit probabilities (Bernoullis)
is the reconstruction cross-entropy:
LIH (x, z)= IH(Bx "Bz )
= −

d
"

k=1

[xk log zk +(1 − xk ) log(1 − zk )] (2)Note that if x is a binary vector, LIH (x, z) is a negative
log-likelihood for the example x, given the Bernoulli
parameters z. Equation 1 with L = LIH can be writtenθ" , θ!" = arg min EEq0 (X) [LIH (X, gθ" (fθ (X)))]

(3)

θ,θ "

where q 0 (X) denotes the empirical distribution associated to our n training inputs. This optimization will
typically be carried out by stochastic gradient descent.

Extracting and Composing Robust Features with Denoising Autoencoders

2.3. The Denoising Autoencoder
To test our hypothesis and enforce robustness to partially destroyed inputs we modify the basic autoencoder we just described. We will now train it to reconstruct a clean “repaired” input from a corrupted, partially destroyed one. This is done by first corrupting
the initial input x to get a partially destroyed versionx̃ by means of a stochastic mapping x̃ ∼ qD (x̃|x). In
our experiments, we considered the following corrupting process, parameterized by the desired proportion νof “destruction”: for each input x, a fixed number νdof components are chosen at random, and their value
is forced to 0, while the others are left untouched. All
information about the chosen components is thus removed from that particuler input pattern, and the autoencoder will be trained to “fill-in” these artificiallyintroduced “blanks”. Note that alternative corrupting
noises could be considered1 . The corrupted input x̃ isthen mapped, as with the basic autoencoder, to a hidden representation y = fθ (x̃) = s(Wx̃+b) from which
we reconstruct a z = gθ" (y) = s(W! y + b! ) (see figure
1 for a schematic representation of the process). As
before the parameters are trained to minimize the average reconstruction error LIH (x, z) = IH(Bx "Bz ) overa training set, i.e. to have z as close as possible to theuncorrupted input x. But the key difference is that z
is now a deterministic function of x̃ rather than x andthus the result of a stochastic mapping of x.
y
gθ!

fθ

LH (x, z)xz

Figure 1. An example x is corrupted to x̃. The autoencoder then maps it to y and attempts to reconstruct x.

Let us define the joint distribution% Y ) = q 0 (X)qD (X|X)δ
%q 0 (X, X,
e (Y )fθ (X)

2.4. Layer-wise Initialization and Fine Tuning
The basic autoencoder has been used as a building
block to train deep networks (Bengio et al., 2007), withthe representation of the k-th layer used as input forthe (k + 1)-th, and the (k + 1)-th layer trained after
the k-th has been trained. After a few layers have been
trained, the parameters are used as initialization for a
network optimized with respect to a supervised training criterion. This greedy layer-wise procedure has
been shown to yield significantly better local minima
than random initialization of deep networks , achieving
better generalization on a number of tasks (Larochelle
et al., 2007).
The procedure to train a deep network using the denoising autoencoder is similar. The only difference is
how each layer is trained, i.e., to minimize the criterion in eq. 5 instead of eq. 3. Note that the corruption process qD is only used during training, but notfor propagating representations from the raw input tohigher-level representations. Note also that when layer
k is trained, it receives as input the uncorrupted output of the previous layers.

3. Relationship to Other ApproachesqDx̃

towards reconstructing the uncorrupted version fromthe corrupted version. Note that in this way, the autoencoder cannot learn the identity, unlike the basic
autoencoder, thus removing the constraint that d! < dor the need to regularize specifically to avoid such atrivial solution.

(4)

where δu (v) puts mass 0 when u &= v. Thus Y is a
% q 0 (X, X,% Y ) is paramdeterministic function of X.
eterized by θ. The objective function minimized bystochastic gradient descent becomes:& #$'%arg min EEq0 (X,X)
LIH X, gθ" (fθ (X))
.
(5)
e
θ,θ "

So from the point of view of the stochastic gradient descent algorithm, in addition to picking an input sample from the training set, we will also produce a random corrupted version of it, and take a gradient step1
The approach we describe and our analysis is not specific to a particular kind of corrupting noise.

Our training procedure for the denoising autoencoder
involves learning to recover a clean input from a corrupted version, a task known as denoising. The problem of image denoising, in particular, has been extensively studied in the image processing community andmany recent developments rely on machine learning
approaches (see e.g. Roth and Black (2005); Elad andAharon (2006); Hammond and Simoncelli (2007)). A
particular form of gated autoencoders has also beenused for denoising in Memisevic (2007). Denoising using autoencoders was actually introduced much earlier (LeCun, 1987; Gallinari et al., 1987), as an alternative to Hopfield models (Hopfield, 1982). Our objective however is fundamentally different from that ofdeveloping a competitive image denoising algorithm.
We investigate explicit robustness to corrupting noise
as a novel criterion guiding the learning of suitable intermediate representations to initialize a deep network.
Thus our corruption+denoising procedure is applied
not only on the input, but also recursively to intermediate representations.

Extracting and Composing Robust Features with Denoising Autoencoders

The approach also bears some resemblance to the wellknown technique of augmenting the training data withstochastically “transformed” patterns. E.g. augmenting a training set by transforming original bitmaps
through small rotations, translations, and scalings is
known to improve final classification performance. In
contrast to this technique our approach does not use
any prior knowledge of image topology, nor does it produce extra labeled examples for supervised training.
We use corrupted patterns in a generic (i.e. not specific to images) unsupervised initialization step, while
the supervised training phase uses the unmodified original data.

ing features. Also note that in section 4.2 we will see
how our learning algorithm for the denoising autoencoder can be viewed as a form of variational inference
in a particular generative model.

There is a well known link between “training withnoise” and regularization: they are equivalent for small
additive noise (Bishop, 1995). By contrast, our corruption process is a large, non-additive, destructionof information. We train autoencoders to ”fill in theblanks”, not merely be smooth functions (regularization). Also in our experience, regularized autoencoders(i.e. with weight decay) do not yield the quantitative
jump in performance and the striking qualitative difference observed in the filters that we get with denoising autoencoders.

The process of mapping a corrupted example to an
uncorrupted one can be visualized in Figure 2, witha low-dimensional manifold near which the data con% that
centrate. We learn a stochastic operator p(X|X)%%
maps an X to an X, p(X|X) = Bg " (fθ (X))
e (X). The
θ
corrupted examples will be much more likely to beoutside and farther from the manifold than the uncor%
rupted ones. Hence the stochastic operator p(X|X)
learns a map that tends to go from lower probability% to high probability points X, generally on
points X% is farther
or near the manifold. Note that when X% should learn to make bigfrom the manifold, p(X|X)
ger steps, to reach the manifold. At the limit we seethat the operator should map even far away points toa small volume near the manifold.

There are also similarities with the work of (Doi et al.,2006) on robust coding over noisy channels. In theirframework, a linear encoder is to encode a clean input
for optimal transmission over a noisy channel to a decoder that reconstructs the input. This work was later
extended to robustness to noise in the input, in a proposal for a model of retinal coding (Doi & Lewicki,
2007). Though some of the inspiration behind ourwork comes from neural coding and computation, ourgoal is not to account for experimental data of neuronal activity as in (Doi & Lewicki, 2007). Also, thenon-linearity of our denoising autoencoder is crucial
for its use in initializing a deep neural network.
It may be objected that, if our goal is to handle missing
values correctly, we could have more naturally defineda proper latent variable generative model, and infer theposterior over the latent (hidden) representation in thepresence of missing inputs. But this usually requires
a costly marginalization2 which has to be carried out
for each new example. By contrast, our approach tries
to learn a fast and robust deterministic mapping fθ
from examples of already corrupted inputs. The burden is on learning such a constrained mapping during
training, rather than on unconstrained inference at use
time. We expect this may force the model to capture
implicit invariances in the data, and result in interest2

as for RBMs, where it is exponential in the number ofmissing values

4. Analysis of Denoising Autoencoders
The above intuitive motivation for the denoising autoencoder was given with the perspective of discovering robust representations. In the following, which canbe skipped without hurting the remainder of the paper,
we propose alternative perspectives on the algorithm.
4.1. Manifold Learning Perspective

The denoising autoencoder can thus be seen as a way
to define and learn a manifold. The intermediate representation Y = f (X) can be interpreted as a coordinate system for points on the manifold (this is mostclear if we force the dimension of Y to be smaller thanthe dimension of X). More generally, one can think of
Y = f (X) as a representation of X which is well suited
to capture the main variations in the data, i.e., on the
manifold. When additional criteria (such as sparsity)
are introduced in the learning model, one can no longer
directly view Y = f (X) as an explicit low-dimensional
coordinate system for points on the manifold, but it
retains the property of capturing the main factors ofvariation in the data.
4.2. Top-down, Generative Model Perspective
In this section we recover the training criterion forour denoising autoencoder (eq. 5) from a generativemodel perspective. Specifically we show that training
the denoising autoencoder as described in section 2.3
is equivalent to maximizing a variational bound on aparticular generative model.
% Y) =
Consider the generative model p(X, X,%
p(Y )p(X|Y )p(X|X) where p(X|Y ) = Bs(W" Y +b" ) andExtracting and Composing Robust Features with Denoising Autoencoders%%p(X|X)= qD (X|X).
p(Y ) is a uniform prior overd"Y ∈ [0, 1] . This defines a generative model with parameter set θ! = {W! , b! }. We will use the previ% Y ) = q 0 (X)qD (X|X)δ
%ously defined q 0 (X, X,
e (Y )fθ (X)(equation 4) as an auxiliary model in the context ofa variational approximation of the log-likelihood of% Note that we abuse notation to make it lighter,p(X).
% and Y for different
and use the same letters X, X
sets of random variables representing the same quantity under different distributions: p or q 0 . Keep inmind that whereas we had the dependency structure% → Y for q or q 0 , we have Y → X → X% for p.
X→X

Since p contains a corruption operation at the last
% to corrupted
generative stage, we propose to fit p(X)training samples. Performing maximum likelihood fit% corresponds to minting for samples drawn from q 0 (X)imizing the cross-entropy, or maximizing%%
H = max{−IH(q 0 (X)"p(
X))}
"
θ

%
= max
{EEq0 (X)e [log p(X)]}.
"

(6)

θ

% be a conditional density, the quanLet q (X, Y |X)&'e% = EE "tity L(q " , X)
log p(X,X,Y ) is a lowere"eq " (X,Y |X)

q (X,Y |X)% since the following can be shown tobound on log p(X)be true for any q " :% = L(q " , X)
% + IDKL (q " (X, Y |X)"p(X,%%log p(X)
Y |X))

Also it is easy to verify that the bound is tight when
% = p(X, Y |X),
% where the IDKL becomes 0.
q " (X, Y |X)% = maxq" L(q " , X),% and
We can thus write log p(X)consequently rewrite equation 6 as%
{EEq0 (X)L(q , X)]}H = maxe [max
"
"
"

q

θ

%
= max
{EEq0 (X)e [L(q , X)]}
" "
"

θ ,q

(7)

gθ( (fθ (x̃))

θ,θ "

(

% Y)
p(X, X,= arg max EEq0 (X,X,Ye ) log%θ,θ "q 0 (X, Y |X)&'%= arg max EEq0 (X,X,Ye ) log p(X, X, Y)

)

θ,θ "

&
'
%
+ EEq0 (X)IH[q 0 (X, Y |X)]e&'%
= arg max EEq0 (X,X,Y
log
p(X,
X,
Y
)
.
e )
θ,θ "

% and θ! only
Note that θ only occurs in Y = fθ (X),
occurs in p(X|Y ). The last line is therefore obtained0% ∝ qD (X|X)q%
because q 0 (X|X)(X) (none of which de!
0%pends on (θ, θ )), and q (Y |X) is deterministic, i.e., its
entropy is constant, irrespective of (θ, θ! ). Hence the% = q 0 (Y |X)q
% 0 (X|X),% does not
entropy of q 0 (X, Y |X)!
vary with (θ, θ ). Finally, following from above, we
obtain our training criterion (eq. 5):
0 %arg max EEq0 (X)e [L(q , X)]θ,θ "%= arg max EEq0 (X,X,Ye )[log[p(Y )p(X|Y )p(X|X)]]θ,θ "= arg max EEq0 (X,X,Ye ) [log p(X|Y )]θ,θ "%= arg max EEq0 (X,X)
e [log p(X|Y = fθ (X))]

& #
$'
%
= arg min EEq0 (X,X)
LIH X, gθ" (fθ (X))
e
θ,θ "

x
x̃

x

Figure 2. Manifold learning perspective. Suppose
training data (×) concentrate near a low-dimensional manifold. Corrupted examples ( ) obtained by applying coreruption process qD (X|X)
will lie farther from the manifold.
e to “project them back” onto
The model learns with p(X|X)
the manifold. Intermediate representation Y can be interpreted as a coordinate system for points on the manifold.

.

0 %arg max{EEq0 (X)e [L(q , X)]}

θ,θ "

x̃

qD (x̃|x)where we moved the maximization outside of the ex% can
pectation because an unconstrained q " (X, Y |X)
in principle perfectly model the conditional distribu% for any X.
% Nowtion needed to maximize L(q " , X)
if we replace the maximization over an unconstrained
q " by the maximization over the parameters θ of ourq 0 (appearing in fθ that maps an x to a y), we get
0 %a lower bound on H: H ≥ maxθ" ,θ {EEq0 (X)e [L(q , X)]}
Maximizing this lower bound, we findwhere the third line is obtained because (θ, θ! )have no influence on EEq0 (X,X,Ye ) [log p(Y )] because
we chose p(Y ) uniform, i.e.
constant, nor on%
EEq0 (X,X)
e [log p(X|X)], and the last line is obtainedby inspection of the definition of LIH in eq. 2, when
% is a B
p(X|Y = fθ (X))
e .
g " (fθ (X))θ

4.3. Other Theoretical Perspectives

Information Theoretic Perspective: Consider%
X ∼ q(X), q unknown, Y = fθ (X).
It can easily
be shown (Vincent et al., 2008) that minimizing the
expected reconstruction error amounts to maximizing

Extracting and Composing Robust Features with Denoising Autoencoders

a lower bound on mutual information I(X; Y ). Denoising autoencoders can thus be justified by the objectivethat Y captures as much information as possible about
X even as Y is a function of corrupted input.
Stochastic Operator Perspective: Extending the
manifold perspective, the denoising autoencoder canalso be seen as corresponding to a semi-parametricmodel from which we can sample (Vincent et al., 2008):*n *% = x̃)qD (x̃|xi ),p(X) = n1 i=1 x̃ p(X|X
where xi is one of the n training examples.

5. Experiments
We performed experiments with the proposed algorithm on the same benchmark of classification problems used in (Larochelle et al., 2007)3 . It contains
different variations of the MNIST digit classification
problem (input dimensionality d = 28 × 28 = 784),with added factors of variation such as rotation (rot),
addition of a background composed of random pixels(bg-rand) or made from patches extracted from a set ofimages (bg-img), or combinations of these factors (rotbg-img). These variations render the problems particularly challenging for current generic learning algorithms. Each problem is divided into a training,validation, and test set (10000, 2000, 50000 examples
respectively). A subset of the original MNIST problem is also included with the same example set sizes(problem basic). The benchmark also contains additional binary classification problems: discriminating
between convex and non-convex shapes (convex), andbetween wide and long rectangles (rect, rect-img).
Neural networks with 3 hidden layers initialized bystacking denoising autoencoders (SdA-3), and finetuned on the classification tasks, were evaluated on all
the problems in this benchmark. Model selection was
conducted following a similar procedure as Larochelle
et al. (2007). Several values of hyper parameters (destruction fraction ν, layer sizes, number of unsupervised training epochs) were tried, combined with early
stopping in the fine tuning phase. For each task, thebest model was selected based on its classification performance on the validation set.
Table 1 reports the resulting classification error on thetest set for the new model (SdA-3), together with theperformance reported in Larochelle et al. (2007)4 for3
All the datasets for these problems are available at
http://www.iro.umontreal.ca/∼lisa/icml2007.
4
Except that rot and rot-bg-img, as reported on the website from which they are available, have been regenerated
since Larochelle et al. (2007), to fix a problem in the initial
data generation process. We used the updated data andcorresponding benchmark results given on this website.

SVMs with Gaussian and polynomial kernels, 1 and 3
hidden layers deep belief network (DBN-1 and DBN-3)
and a 3 hidden layer deep network initialized by stacking basic autoencoders (SAA-3). Note that SAA-3 isequivalent to a SdA-3 with ν = 0% destruction. As canbe seen in the table, the corruption+denoising training works remarkably well as an initialization step, andin most cases yields significantly better classification
performance than basic autoencoder stacking with nonoise. On all but one task the SdA-3 algorithm performs on par or better than the best other algorithms,
including deep belief nets. Due to space constraints,
we do not report all selected hyper-parameters in thetable (only showing ν). But it is worth mentioning
that, for the majority of tasks, the model selection
procedure chose best performing models with an overcomplete first hidden layer representation (typicallyof size 2000 for the 784-dimensional MNIST-derived
tasks). This is very different from the traditional “bottleneck” autoencoders, and made possible by our denoising training procedure. All this suggests that the
proposed procedure was indeed able to produce more
useful feature detectors.
Next, we wanted to understand qualitatively the effect of the corruption+denoising training. To this end
we display the filters obtained after initial training of
the first denoising autoencoder on MNIST digits. Figure 3 shows a few of these filters as little image patches,for different noise levels. Each patch corresponds to arow of the learnt weight matrix W, i.e. the incoming
weights of one of the hidden layer neurons. The beneficial effect of the denoising training can clearly be seen.
Without the denoising procedure, many filters appear
to have learnt no interesting feature. They look likethe filters obtained after random initialization. But
when increasing the level of destructive corruption, an
increasing number of filters resemble sensible featuredetectors. As we move to higher noise levels, we observe a phenomenon that we expected: filters becomeless local, they appear sensitive to larger structures
spread out across more input dimensions.

6. Conclusion and Future Work
We have introduced a very simple training principle
for autoencoders, based on the objective of undoing acorruption process. This is motivated by the goal oflearning representations of the input that are robust to
small irrelevant changes in input. We also motivatedit from a manifold learning perspective and gave aninterpretation from a generative model perspective.
This principle can be used to train and stack autoencoders to initialize a deep neural network. A series

Extracting and Composing Robust Features with Denoising Autoencoders

Table 1. Comparison of stacked denoising autoencoders (SdA-3) with other models.
Test error rate on all considered classification problems is reported together with a 95% confidence interval. Best performer
is in bold, as well as those for which confidence intervals overlap. SdA-3 appears to achieve performance superior or
equivalent to the best other model on all problems except bg-rand. For SdA-3, we also indicate the fraction ν of destroyed
input components, as chosen by proper model selection. Note that SAA-3 is equivalent to SdA-3 with ν = 0%.

Datasetbasicrot
bg-randbg-imgrot-bg-img
rect
rect-img
convex

SVMrbf3.03±0.15
11.11±0.28
14.58±0.31
22.61±0.37
55.18±0.44
2.15±0.13
24.04±0.37
19.13±0.34(a) No destroyed inputs

SVMpoly
3.69±0.17
15.42±0.32
16.62±0.33
24.01±0.37
56.41±0.43
2.15±0.13
24.05±0.37
19.82±0.35

DBN-1
3.94±0.17
14.69±0.31
9.80±0.26
16.15±0.32
52.21±0.44
4.71±0.19
23.69±0.37
19.92±0.35

SAA-3
3.46±0.16
10.30±0.27
11.28±0.28
23.00±0.37
51.93±0.44
2.41±0.13
24.05±0.37
18.41±0.34(b) 25% destruction(d) Neuron A (0%, 10%, 20%, 50% destruction)

DBN-3
3.11±0.15
10.30±0.27
6.73±0.22
16.31±0.32
47.39±0.44
2.60±0.14
22.50±0.37
18.63±0.34

SdA-3 (ν)
2.80±0.14 (10%)
10.29±0.27 (10%)
10.38±0.27 (40%)
16.68±0.33 (25%)
44.49±0.44 (25%)
1.99±0.12 (10%)
21.59±0.36 (25%)
19.06±0.34 (10%)(c) 50% destruction(e) Neuron B (0%, 10%, 20%, 50% destruction)

Figure 3. Filters obtained after training the first denoising autoencoder.
(a-c) show some of the filters obtained after training a denoising autoencoder on MNIST samples, with increasing
destruction levels ν. The filters at the same position in the three images are related only by the fact that the autoencoders
were started from the same random initialization point.
(d) and (e) zoom in on the filters obtained for two of the neurons, again for increasing destruction levels.
As can be seen, with no noise, many filters remain similarly uninteresting (undistinctive almost uniform grey patches).
As we increase the noise level, denoising training forces the filters to differentiate more, and capture more distinctive
features. Higher noise levels tend to induce less local filters, as expected. One can distinguish different kinds of filters,from local blob detectors, to stroke detectors, and some full character detectors at the higher noise levels.

Extracting and Composing Robust Features with Denoising Autoencodersof image classification experiments were performed toevaluate this new training principle. The empirical results support the following conclusions: unsupervised
initialization of layers with an explicit denoising criterion helps to capture interesting structure in the input
distribution. This in turn leads to intermediate representations much better suited for subsequent learning tasks such as supervised classification. It is possible that the rather good experimental performance ofDeep Belief Networks (whose layers are initialized as
RBMs) is partly due to RBMs encapsulating a similar form of robustness to corruption in the representations they learn, possibly because of their stochastic nature which introduces noise in the representation
during training. Future work inspired by this observation should investigate other types of corruption process, not only of the input but of the representation
itself as well.

Acknowledgments
We thank the anonymous reviewers for their usefulcomments that helped improved the paper. We arealso very grateful for financial support of this work by
NSERC, MITACS and CIFAR.

References
Bengio, Y. (2007). Learning deep architectures for AI(Technical Report 1312). Université de Montréal, dept.
IRO.
Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H.
(2007). Greedy layer-wise training of deep networks.
Advances in Neural Information Processing Systems 19(pp. 153–160). MIT Press.
Bengio, Y., & Le Cun, Y. (2007). Scaling learning algorithms towards AI. In L. Bottou, O. Chapelle, D. DeCoste and J. Weston (Eds.), Large scale kernel machines.
MIT Press.
Bishop, C. M. (1995). Training with noise is equivalent totikhonov regularization. Neural Computation, 7, 108–
116.
Doi, E., Balcan, D. C., & Lewicki, M. S. (2006). A theoretical analysis of robust coding over noisy overcomplete
channels. In Y. Weiss, B. Schölkopf and J. Platt (Eds.),
Advances in neural information processing systems 18,
307–314. Cambridge, MA: MIT Press.
Doi, E., & Lewicki, M. S. (2007). A theory of retinal population coding. NIPS (pp. 353–360). MIT Press.
Elad, M., & Aharon, M. (2006). Image denoising via sparseand redundant representations over learned dictionaries.
IEEE Transactions on Image Processing, 15, 3736–3745.
Gallinari, P., LeCun, Y., Thiria, S., & Fogelman-Soulie, F.
(1987). Memoires associatives distribuees. Proceedingsof COGNITIVA 87. Paris, La Villette.

Hammond, D., & Simoncelli, E. (2007). A machine learning
framework for adaptive combination of signal denoising
methods. 2007 International Conference on Image Processing (pp. VI: 29–32).
Hinton, G. (1989). Connectionist learning procedures. Artificial Intelligence, 40, 185–234.
Hinton, G., & Salakhutdinov, R. (2006). Reducing thedimensionality of data with neural networks. Science,
313, 504–507.
Hinton, G. E., Osindero, S., & Teh, Y. (2006). A fast learning algorithm for deep belief nets. Neural Computation,
18, 1527–1554.
Hopfield, J. (1982). Neural networks and physical systems
with emergent collective computational abilities. Proceedings of the National Academy of Sciences, USA, 79.
Larochelle, H., Erhan, D., Courville, A., Bergstra, J., &Bengio, Y. (2007). An empirical evaluation of deep architectures on problems with many factors of variation.
Proceedings of the 24 th International Conference on Machine Learning (ICML’2007) (pp. 473–480).
LeCun, Y. (1987).
Modèles connexionistes del’apprentissage.
Doctoral dissertation, Universitéde Paris VI.
Lee, H., Ekanadham, C., & Ng, A. (2008). Sparse deep belief net model for visual area V2. In J. Platt, D. Koller,
Y. Singer and S. Roweis (Eds.), Advances in neural information processing systems 20. Cambridge, MA: MIT
Press.
McClelland, J., Rumelhart, D., & the PDP Research Group (1986). Parallel distributed processing:
Explorations in the microstructure of cognition, vol. 2.
Cambridge: MIT Press.
Memisevic, R. (2007). Non-linear latent factor models forrevealing structure in high-dimensional data. Doctoral
dissertation, Departement of Computer Science, University of Toronto, Toronto, Ontario, Canada.
Ranzato, M., Boureau, Y.-L., & LeCun, Y. (2008). Sparsefeature learning for deep belief networks. In J. Platt,
D. Koller, Y. Singer and S. Roweis (Eds.), Advances inneural information processing systems 20. Cambridge,
MA: MIT Press.
Ranzato, M., Poultney, C., Chopra, S., & LeCun, Y.
(2007). Efficient learning of sparse representations withan energy-based model. Advances in Neural Information
Processing Systems (NIPS 2006). MIT Press.
Roth, S., & Black, M. (2005). Fields of experts: a framework for learning image priors. IEEE Conference on
Computer Vision and Pattern Recognition (pp. 860–
867).
Utgoff, P., & Stracuzzi, D. (2002). Many-layered learning.
Neural Computation, 14, 2497–2539.
Vincent, P., Larochelle, H., Bengio, Y., & Manzagol, P.-A.
(2008). Extracting and composing robust features withdenoising autoencoders (Technical Report 1316). Université de Montréal, dept. IRO.