Saturating Auto-Encoder
Authors: Ross Goroshin, Yann LeCun
ICLR 2013

Abstract: We introduce a simple new regularizer for auto-encoders whose hidden-unit activation functions contain at least one zero-gradient (saturated) region. This regularizer explicitly encourages activations in the saturated region(s) of the corresponding activation function. We call these Saturating Auto-Encoders (SATAE). We show that the saturation regularizer explicitly limits the SATAE's ability to reconstruct inputs which are not near the data manifold. Furthermore, we show that a wide variety of features can be learned when different activation functions are used. Finally, connections are established with the Contractive and Sparse Auto-Encoders.

Saturating Auto-Encoders
Rostislav Goroshin∗
Courant Institute of Mathematical Science
New York University
goroshin@cs.nyu.edu
Yann LeCun
Courant Institute of Mathematical Science
New York University
yann@cs.nyu.edu
Abstract
We introduce a simple new regularizer for auto-encoders whose hidden-unit ac-
tivation functions contain at least one zero-gradient (saturated) region. This reg-
ularizer explicitly encourages activations in the saturated region(s) of the corre-
sponding activation function. We call these Saturating Auto-Encoders (SATAE).
We show that the saturation regularizer explicitly limits the SATAE’s ability to
reconstruct inputs which are not near the data manifold. Furthermore, we show
that a wide variety of features can be learned when different activation functions
are used. Finally, connections are established with the Contractive and Sparse
Auto-Encoders.
1 Introduction
An auto-encoder is a conceptually simple neural network used for obtaining useful data rep-
resentations through unsupervised training. It is composed of an encoder which outputs a
hidden (or latent) representation and a decoder which attempts to reconstruct the input using
the hidden representation as its input. Training consists of minimizing a reconstruction cost
such as L2 error. However this cost is merely a proxy for the true objective: to obtain a useful
latent representation. Auto-encoders can implement many dimensionality reduction techniques
such as PCA and Sparse Coding (SC) [5] [6] [7]. This makes the study of auto-encoders very
appealing from a theoretical standpoint. In recent years, renewed interest in auto-encoders net-
works has mainly been due to their empirical success in unsupervised feature learning [1] [2] [3] [4].
When minimizing only reconstruction cost, the standard auto-encoder does not typically learn any
meaningful hidden representation of the data. Well known theoretical and experimental results
show that a linear auto-encoder with trainable encoding and decoding matrices, We and Wd re-
spectively, learns the identity function if We and Wd are full rank or over-complete. The linear
auto-encoder learns the principle variance directions (PCA) if We and Wd are rank deﬁcient [5].
It has been observed that other representations can be obtained by regularizing the latent repre-
sentation. This approach is exempliﬁed by the Contractive and Sparse Auto-Encoders [3] [1] [2].
Intuitively, an auto-encoder with limited capacity will focus its resources on reconstructing portions
of the input space in which data samples occur most frequently. From an energy based perspective,
auto-encoders achieve low reconstruction cost in portions of the input space with high data density
(recently, [8] has examined this perspective in depth). If the data occupies some low dimensional
manifold in the higher dimensional input space then minimizing reconstruction error achieves low
energy on this manifold. Useful latent state regularizers raise the energy of points that do not lie
on the manifold, thus playing an analogous role to minimizing the partition function in maximum
likelihood models. In this work we introduce a new type of regularizer that does this explicitly for
∗The authors thank Joan Bruna and David Eigen for their useful suggestions and comments.
1
arXiv:1301.3577v3  [cs.LG]  20 Mar 2013
auto-encoders with a non-linearity that contains at least one ﬂat (zero gradient) region. We show ex-
amples where this regularizer and the choice of nonlinearity determine the feature set that is learned
by the auto-encoder.
2 Latent State Regularization
Several auto-encoder variants which regularize their latent states have been proposed, they include
the sparse auto-encoder and the contractive auto-encoder [1] [2] [3]. The sparse auto-encoder in-
cludes an over-complete basis in the encoder and imposes a sparsity inducing (usually L1) penalty
on the hidden activations. This penalty prevents the auto-encoder from learning to reconstruct all
possible points in the input space and focuses the expressive power of the auto-encoder on repre-
senting the data-manifold. Similarly, the contractive auto-encoder avoids trivial solutions by intro-
ducing an auxiliary penalty which measures the square Frobenius norm of the Jacobian of the latent
representation with respect to the inputs. This encourages a constant latent representation except
around training samples where it is counteracted by the reconstruction term. It has been noted in [3]
that these two approaches are strongly related. The contractive auto-encoder explicitly encourages
small entries in the Jacobian, whereas the sparse auto-encoder is encouraged to produce mostly zero
(sparse) activations which can be designed to correspond to mostly ﬂat regions of the nonlinearity,
thus also yielding small entries in the Jacobian.
2.1 Saturating Auto-Encoder through Complementary Nonlinearities
Our goal is to introduce a simple new regularizer which explicitly raises reconstruction error for
inputs not near the data manifold. Consider activation functions with at least one ﬂat region; these
include shrink, rectiﬁed linear, and saturated linear (Figure 1). Auto-encoders with such nonlineari-
ties lose their ability to accurately reconstruct inputs which produce activations in the zero-gradient
regions of their activation functions. Let us denote the auto-encoding function xr = G(x,W), x
being the input, W the trainable parameters in the auto-encoder, and xr the reconstruction. One can
deﬁne an energy surface through the reconstruction error:
EW (x) = ||x−G(x,W)||2
Let’s imagine that G has been trained to produce a low reconstruction error at a particular data
point x∗. If Gis constant when xvaries along a particular direction v, then the energy will grow
quadratically along that particular direction as xmoves away from x∗. If Gis trained to produce
low reconstruction errors on a set of samples while being subject to a regularizer that tries to make
it constant in as many directions as possible, then the reconstruction energy will act as a contrast
function that will take low values around areas of high data density and larger values everywhere
else (similarly to a negative log likelihood function for a density estimator).
The proposed auto-encoder is a simple implementation of this idea. Using the notation W =
{We,Be,Wd,Bd}, the auto-encoder function is deﬁned as
G(x,W) = WdF(Wex+ Be) + Bd
where We, Be, Wd, and Bd are the encoding matrix, encoding bias, decoding matrix, and decoding
bias, respectively, and F is the vector function that applies the scalar function f to each of its
components. f will be designed to have ”ﬂat spots”, i.e. regions where the derivative is zero (also
referred to as the saturation region).
The loss function minimized by training is the sum of the reconstruction energy EW (x) = ||x−
G(x,W)||2 and a term that pushes the components ofWex+ Be towards the ﬂat spots of f. This is
performed through the use of a complementary functionfc, associated with the non-linearity f(z).
The basic idea is to design fc(z) so that its value corresponds to the distance of z to one of the
ﬂat spots of f(z). Minimizing fc(z) will push z towards the ﬂat spots of f(z). With this in mind,
we introduce a penalty of the form fc(∑d
j=1 We
ijxj + be
i ) which encourages the argument to be
in the saturation regime of the activation function ( f). We refer to auto-encoders which include
this regularizer as Saturating Auto-Encoders (SATAEs). For activation functions with zero-gradient
regime(s) the complementary nonlinearity (fc) can be deﬁned as the distance to the nearest saturation
region. Speciﬁcally, let S = {z|f′(z) = 0}then we deﬁne fc(z) as:
2
Figure 1: Three nonlinearities (top) with their associated complementary regularization func-
tions(bottom).
fc(z) = inf
z′∈S
|z−z′|. (1)
Figure 1 shows three activation functions and their associated complementary nonlinearities. The
complete loss to be minimized by a SATAE with nonlinearityf is:
L=
∑
x∈D
1
2∥x−
(
WdF(Wex+ Be) + Bd)
∥2 + α
dh∑
i=1
fc(We
i x+ be
i ), (2)
where dh denotes the number of hidden units. The hyper-parameterαregulates the trade-off between
reconstruction and saturation.
3 Effect of the Saturation Regularizer
We will examine the effect of the saturation regularizer on auto-encoders with a variety of activation
functions. It will be shown that the choice of activation function is a signiﬁcant factor in determining
the type of basis the SATAE learns. First, we will present results on toy data in two dimensions
followed by results on higher dimensional image data.
3.1 Visualizing the Energy Landscape
Given a trained auto-encoder the reconstruction error can be evaluated for a given input x. For
low-dimensional spaces (Rn, where n ≤3) we can evaluate the reconstruction error on a regular
grid in order to visualize the portions of the space which are well represented by the auto-encoder.
More speciﬁcally we can compute E(x) = 1
2 ∥x−xr∥2 for all xwithin some bounded region of
the input space. Ideally, the reconstruction energy will be low for all xwhich are in the training set
and high elsewhere. Figures 2 and 3 depict the resulting reconstruction energy for inputs x ∈R2,
and −1 ≤xi ≤1. Black corresponds to low reconstruction energy. The training data consists of
a one dimensional manifold shown overlain in yellow. Figure 2 shows a toy example for a SATAE
which uses ten basis vectors and a shrink activation function. Note that adding the saturation regu-
larizer decreases the volume of the space which is well reconstructed, however good reconstruction
is maintained on or near the training data manifold. The auto-encoder in Figure 3 contains two
3
Figure 2: Energy surfaces for unregularized (left), and regularized (right) solutions obtained on
SATAE-shrink and 10 basis vectors. Black corresponds to low reconstruction energy. Training
points lie on a one-dimensional manifold shown in yellow.
Figure 3: SATAE-SL toy example with two basis elements. Top Row: three randomly initialized so-
lutions obtained with no regularization. Bottom Row: three randomly initialized solutions obtained
with regularization.
encoding basis vectors (red), two decoding basis vectors (green), and uses a saturated-linear activa-
tion function. The encoding and decoding bases are unconstrained. The unregularized auto-encoder
learns an orthogonal basis with a random orientation. The region of the space which is well recon-
structed corresponds to the outer product of the linear regions of two activation functions; beyond
that the error increases quadratically with the distance. Including the saturation regularizer induces
the auto-encoder basis to align with the data and to operate in the saturation regime at the extreme
points of the training data, which limits the space which is well reconstructed. Note that because the
encoding and decoding weights are separate and unrestricted, the encoding weights were scaled up
to effectively reduce the width of the linear regime of the nonlinearity.
3.2 SATAE-shrink
Consider a SATAE with a shrink activation function and shrink parameter λ. The corresponding
complementary nonlinearity, derived using Equation 1 is given by:
shrinkc(x) =
{abs(x), |x|>λ
0, elsewhere .
Note that shrinkc(Wex+ be) = abs(shrink(Wex+ be)), which corresponds to an L1 penalty
on the activations. Thus this SATAE is equivalent to a sparse auto-encoder with a shrink activation
function. Given the equivalence to the sparse auto-encoder we anticipate the same scale ambiguity
which occurs with L1 regularization. This ambiguity can be avoided by normalizing the decoder
weights to unit norm. It is expected that the SATAE-shrink will learn similar features to those
obtained with a sparse auto-encoder, and indeed this is what we observe. Figure 4(c) shows the
decoder ﬁlters learned by an auto-encoder with shrink nonlinearity trained on gray-scale natural
image patches. One can recognize the expected Gabor-like features when the saturation penalty is
activated. When trained on the binary MNIST dataset the learned basis is comprised of portions of
digits and strokes. Nearly identical results are obtained with a SATAE which uses a rectiﬁed-linear
4
activation function. This is because a rectiﬁed-linear function with an encoding bias behaves as a
positive only shrink function, similarly the complementary function is equivalent to a positive only
L1 penalty on the activations.
3.3 SATAE-saturated-linear
Unlike the SATAE-shrink, which tries to compress the data by minimizing the number of active
elements; the SATAE saturated-linear (SATAE-SL) tries to compress the data by encouraging the
latent code to be as close to binary as possible. Without a saturation penalty this auto-encoder learns
to encode small groups of neighboring pixels. More precisely, the auto-encoder learns the identity
function on all datasets. An example of such a basis is shown in Figure 4(b). With this basis the
auto-encoder can perfectly reconstruct any input by producing small activations which stay within
the linear region of the nonlinearity. Introducing the saturation penalty does not have any effect
when training on binary MNIST. This is because the scaled identity basis is a global minimizer of
Equation 2 for the SATAE-SL on any binary dataset. Such a basis can perfectly reconstruct any
binary input while operating exclusively in the saturated regions of the activation function, thus
incurring no saturation penalty. On the other hand, introducing the saturation penalty when training
on natural image patches induces the SATAE-SL to learn a more varied basis (Figure 4(d)).
3.4 Experiments on CIFAR-10
SATAE auto-encoders with 100 and 300 basis elements were trained on the CIFAR-10 dataset,
which contains small color images of objects from ten categories. In all of our experiments the auto-
encoders were trained by progressively increasing the saturation penalty (details are provided in the
next section). This allowed us to visually track the effect of the saturation penalty on individual
basis elements. Figure 4(e)-(f) shows the basis learned by SATAE-shrink with small and large
saturation penalty, respectively. Increasing the saturation penalty has the expected effect of reducing
the number of nonzero activations. As the saturation penalty increases, active basis elements become
responsible for reconstructing a larger portion of the input. This induces the basis elements to
become less spatially localized. This effect can be seen by comparing corresponding ﬁlters in Figure
4(e) and (f). Figures 4(g)-(h) show the basis elements learned by SATAE-SL with small and large
saturation penalty, respectively. The basis learned by SATAE-SL with a small saturation penalty
resembles the identity basis, as expected (see previous subsection). Once the saturation penalty
is increased small activations become more heavily penalized. To increase their activations the
encoding basis elements may increase in magnitude or align themselves with the input. However, if
the encoding and decoding weights are tied (or ﬁxed in magnitude) then reconstruction error would
increase if the weights were merely scaled up. Thus the basis elements are forced to align with the
data in a way that also facilitates reconstruction. This effect is illustrated in Figure 5 where ﬁlters
corresponding to progressively larger values of the regularization parameter are shown. The top
half of the ﬁgure shows how an element from the identity basis ( α= 0.1) transforms to a localized
edge (α= 0.5). The bottom half of the ﬁgure shows how a localized edge ( α= 0.5) progressively
transforms to a template of a horse (α= 1).
4 Experimental Details
Because the regularizer explicitly encourages activations in the zero gradient regime of the nonlin-
earity, many encoder basis elements would not be updated via back-propagation through the non-
linearity if the saturation penalty were large. In order to allow the basis elements to deviate from
their initial random states we found it necessary to progressively increase the saturation penalty. In
our experiments the weights obtained at a minimum of Equation 2 for a smaller value of α were
used to initialize the optimization for a larger value of α. Typically, the optimization began with
α = 0 and was progressively increased to α = 1 in steps of 0.1. The auto-encoder was trained for
30 epochs at each value of α. This approach also allowed us to track the evolution of basis elements
as a function of α(Figure 5). In all experiments data samples were normalized by subtracting the
mean and dividing by the standard deviation of the dataset. The auto-encoders used to obtain the
results shown in Figure 4 (a),(c)-(f) used 100 basis elements, others used 300 basis elements. In-
creasing the number of elements in the basis did not have a strong qualitative effect except to make
the features represented by the basis more localized. The decoder basis elements of the SATAEs with
5
(a)
 (b)
 (c)
 (d)
(e)
 (f)
 (g)
 (h)
Figure 4: Basis elements learned by the SATAE using different nonlinearities on: 28x28 binary
MNIST digits, 12x12 gray scale natural image patches, and CIFAR-10. (a) SATAE-shrink trained on
MNIST, (b) SATAE-saturated-linear trained on MNIST, (c) SATAE-shrink trained on natural image
patches, (d) SATAE-saturated-linear trained on natural image patches, (e)-(f) SATAE-shrink trained
on CIFAR-10 withα= 0.1 and α= 0.5, respectively, (g)-(h) SATAE-SL trained on CIFAR-10 with
α= 0.1 and α= 0.6, respectively. 6
Figure 5: Evolution of two ﬁlters with increasing saturation regularization for a SATAE-SL trained
on CIFAR-10. Filters corresponding to larger values of α were initialized using the ﬁlter corre-
sponding to the previous α. The regularization parameter was varied from 0.1 to 0.5 (left to right)
in the top ﬁve images and 0.5 to 1 in the bottom ﬁve
shrink and rectiﬁed-linear nonlinearities were reprojected to the unit sphere after every 10 stochastic
gradient updates. The SATAEs which used saturated-linear activation function were trained with
tied weights. All results presented were obtained using stochastic gradient descent with a constant
learning rate of 0.05.
5 Discussion
In this work we have introduced a general and conceptually simple latent state regularizer. It was
demonstrated that a variety of feature sets can be obtained using a single framework. The utility of
these features depend on the application. In this section we extend the deﬁnition of the saturation
regularizer to include functions without a zero-gradient region. The relationship of SATAEs with
other regularized auto-encoders will be discussed. We conclude with a discussion on future work.
5.1 Extension to Differentiable Functions
We would like to extend the saturation penalty deﬁnition (Equation 1) to differentiable functions
without a zero-gradient region. An appealing ﬁrst guess for the complimentary function is some
positive function of the ﬁrst derivative, fc(x) = |f′(x)|for instance. This may be an appropriate
choice for monotonic activation functions which have their lowest gradient regions at the extrema
(e.g. sigmoids). However some activation functions may contain regions of small or zero gradient
which have negligible extent, at the extrema for instance. We would like our deﬁnition of the com-
plimentary function to not only measure the local gradient in some region, but to also measure it’s
extent. For this purpose we employ the concept of average variation over a ﬁnite interval. We deﬁne
the average variation of f at xin the positive and negative directions at scale l, respectively as:
∆+
l f(x) = 1
l
∫ x+l
x
|f′(u)|du= |f′(x)|∗Π+
l (x)
∆−
l f(x) = 1
l
∫ x
x−l
|f′(u)|du= |f′(x)|∗Π−
l (x).
Where ∗denotes the continuous convolution operator. Π+
l (x) and Π−
l (x) are uniform averaging
kernels in the positive and negative directions, respectively. Next, deﬁne a directional measure of
variation of f by integrating the average variation at all scales.
7
Figure 6: Illustration of the complimentary function ( fc) as deﬁned by Equation 3 for a non-
monotonic activation function (f). The absolute derivative of f is shown for comparison.
M+f(x) =
∫ +∞
0
∆+
l f(x)w(l)dl=
[∫ +∞
0
w(l)Π+
l (x)dl
]
∗|f′(x)|
M−f(x) =
∫ +∞
0
∆−
l f(x)w(l)dl=
[∫ +∞
0
w(l)Π−
l (x)dl
]
∗|f′(x)|.
Where w(l) is chosen to be a sufﬁciently fast decreasing function of l to insure convergence of
the integral. The integral with which |f′(x)|is convolved in the above equation evaluates to some
decreasing function of xfor Π+ with support x≥0. Similarly, the integral involving Π−evaluates
to some increasing function of x with support x ≤0. This function will depend on w(l). The
functions M+f(x) and M−f(x) measure the average variation off(x) at all scales lin the positive
and negative direction, respectively. We deﬁne the complimentary functionfc(x) as:
fc(x) = min(M+f(x),M−f(x)). (3)
An example of a complimentary function deﬁned using the above formulation is shown in Figure 6.
Whereas |f′(x)|is minimized at the extrema off, the complimentary function only plateaus at these
locations.
5.2 Relationship with the Contractive Auto-Encoder
Let hi be the output of theith hidden unit of a single-layer auto-encoder with point-wise nonlinearity
f(·). The regularizer imposed by the contractive auto-encoder (CAE) can be expressed as follows:
∑
ij
(∂hi
∂xj
)2
=
dh∑
i

f′(
d∑
j=1
We
ijxj + bi)2∥We
i ∥2

,
where x is a d-dimensional data vector, f′(·) is the derivative of f(·), bi is the bias of the ith
encoding unit, andWe
i denotes the ith row of the encoding weight matrix. The ﬁrst term in the above
equation tries to adjust the weights so as to push the activations into the low gradient (saturation)
regime of the nonlinearity, but is only deﬁned for differentiable activation functions. Therefore the
CAE indirectly encourages operation in the saturation regime. Computing the Jacobian, however,
can be cumbersome for deep networks. Furthermore, the complexity of computing the Jacobian is
O(d×dh), although a more efﬁcient implementation is possible [3], compared to the O(dh) for the
saturation penalty.
8
5.3 Relationship with the Sparse Auto-Encoder
In Section 3.2 it was shown that SATAEs with shrink or rectiﬁed-linear activation functions are
equivalent to a sparse auto-encoder. Interestingly, the fact that the saturation penalty happens to
correspond to L1 regularization in the case of SATAE-shrink agrees with the ﬁndings in [7]. In their
efforts to ﬁnd an architecture to approximate inference in sparse coding, Gregor et al. found that
the shrink function is particularly compatible with L1 minimization. Equivalence to sparsity only
for some activation functions suggests that SATAEs are a generalization of sparse auto-encoders.
Like the sparsity penalty, the saturation penalty can be applied at any point in a deep network for the
same computational cost. However, unlike the sparsity penalty the saturation penalty is adapted to
the nonlinearity of the particular layer to which it is applied.
5.4 Future Work
We intend to experimentally demonstrate that the representations learned by SATAEs are useful as
features for learning common tasks such as classiﬁcation and denoising. We will also address several
open questions, namely: (i) how to select (or learn) the width parameter (λ) of the nonlinearity, and
(ii) how to methodically constrain the weights. We will also explore SATAEs that use a wider class
of non-linearities and architectures.
References
[1] Marc’Aurelio Ranzato, Christopher Poultney, Sumit Chopra and Yann LeCun. Efﬁcient Learn-
ing of Sparse Representations with an Energy- Based Model, in J. Platt et al. (Eds),Advances in
Neural Information Processing Systems (NIPS 2006), 19, MIT Press, 2006.
[2] Marc’Aurelio Ranzato, Fu-Jie Huang, Y-Lan Boureau and Yann LeCun: Unsupervised Learn-
ing of Invariant Feature Hierarchies with Applications to Object Recognition, Proc. Computer
Vision and Pattern Recognition Conference (CVPR’07), IEEE Press, 2007
[3] Rifai, S. and Vincent, P. and Muller, X. and Glorot, X. and Bengio, Y . Contractive auto-encoders:
Explicit invariance during feature extraction, Proceedings of the Twenty-eight International
Conference on Machine Learning, ICML 2011
[4] P. Vincent, H. Larochelle, Y . Bengio, P.A. Manzagol. Extracting and Composing Robust Fea-
tures with Denoising Autoencoders Proceedings of the 25th International Conference on Ma-
chine Learning (ICML’2008), 2008.
[5] R.O. Duda, P.E. Hart, and D.G. Stork, Pattern Classiﬁcation, New York: John Wiley & Sons,
2001, pp. xx + 654, ISBN: 0-471-05669-3
[6] Olhausen, Bruno A.; Field, David J. (1997). Sparse Coding with an Overcomplete Basis Set: A
Strategy Employed by V1?. Vision Research 37 (23): 3311-3325.
[7] Karol Gregor and Yann LeCun: Learning Fast Approximations of Sparse Coding, Proc. Inter-
national Conference on Machine learning (ICML’10), 2010
[8] Guillaume Alain and Yoshua Bengio, What Regularized Auto-Encoders Learn from the Data
Generating Distribution. arXiv:1211.4246v3 [cs.LG]
9


Peer review:
[Review]:
This is a cool investigation in a direction that I find fascinating, and I only have two remarks about minor points made in the paper.

* Regarding the energy-based interpretation (that reconstruction error can be thought of as an energy function associated with an estimated probability function), there was a recent result which surprised me and challenges that view. In http://arxiv.org/abs/1211.4246 (What Regularized Auto-Encoders Learn from the Data Generating Distribution), Guillaume Alain and I found that denoising and contractive auto-encoders (where we penalize the Jacobian of the encoder-decoder function r(x)=decode(encode(x))) estimate the *score* of the data generating function in the vector r(x)-x (I should also mention Vincent 2011 Neural Comp. with a similar earlier result for a particular form of denoising auto-encoder where there is a well-defined energy function).  So according to these results,  the reconstruction error ||r(x)-x||^2 would be the magnitude of the score (derivative of energy wrt input). This is quite different from the energy itself, and it would suggest that the reconstruction error would be near zero both at a *minimum* of the energy (near training examples) AND at a *maximum* of the energy (e.g. near peaks that separate valleys of the energy). We have actually observed that empirically in toy problems where one can visualize the score in 2D.

* Regarding the comparison in section 5.1 with the contractive auto-encoder, I believe that there is a correct but somewhat misleading statement. It says that the contractive penalty costs O(d * d_h) to compute whereas the saturating penalty only costs O(d_h) to compute. This is true, but since computing h in the first place also costs O(d * d_h) the overhead of the contractive penalty is small (it basically doubles the computational cost, which is much less problematic than multiplying it by d as the remark could lead a naive reader to believe).