Switched linear encoding with rectified linear autoencoders
Authors: Leif Johnson, Craig Corcoran
ICLR 2013

Abstract: Several recent results in machine learning have established formal connections between autoencoders---artificial neural network models that attempt to reproduce their inputs---and other coding models like sparse coding and K-means. This paper explores in depth an autoencoder model that is constructed using rectified linear activations on its hidden units. Our analysis builds on recent results to further unify the world of sparse linear coding models. We provide an intuitive interpretation of the behavior of these coding models and demonstrate this intuition using small, artificial datasets with known distributions.

Switched linear coding with rectiﬁed linear
autoencoders
Leif Johnson Craig Corcoran
Computer Science Department
The University of Texas at Austin
Austin, TX 78701
{leif,ccor}@cs.utexas.edu
Abstract
Several recent results in machine learning have established formal connections be-
tween autoencoders—artiﬁcial neural network models that attempt to reproduce
their inputs—and other coding models like sparse coding and K-means. This pa-
per explores in depth an autoencoder model that is constructed using rectiﬁed lin-
ear activations on its hidden units. Our analysis builds on recent results to further
unify the world of sparse linear coding models. We provide an intuitive interpre-
tation of the behavior of these coding models and demonstrate this intuition using
small, artiﬁcial datasets with known distributions.
1 Introduction
Large quantities of natural data are now commonplace in the digital world—images, videos, and
sounds are all relatively cheap and easy to obtain. In comparison, labels for these datasets (e.g.,
answers to questions like, “Is this an image of a cow or a horse?”) remain relatively expensive
and difﬁcult to assemble. In addition, even when labeled data are available for a given task, there
are often only a few bits of information in the labels, while the unlabeled data can easily contain
millions of bits. Finally, most collections of data from the natural world also seem to be distributed
non-uniformly in the space of all possible inputs, suggesting that there is some underlying structure
inherent in a particular dataset, independent of labels or task. Furthermore, many unsupervised
learning approaches assume that natural data even lie on a relatively continuous low-dimensional
manifold of the available space, which provides further structure that can be captured in a model.
For these reasons, much recent work in machine learning has focused on unsupervised modeling of
large, easy to obtain datasets.
An unsupervised model attempts to create a compact representation of the structure of a dataset.
Some models are designed to learn a set of “basis functions” that can be combined linearly to create
the observed data. Once such a set has been learned, this representation is often useful in other tasks
like classiﬁcation or recognition (but see [4] for interesting analysis). In addition to learning useful
representations, many models that have resulted from this line of work have also been shown to have
interesting theoretical ties with neuroscience and information theory (e.g., [12]).
In this paper, we ﬁrst synthesize recent results from unsupervised machine learning, with a particular
focus on neural networks and sparse coding. We then explore in detail the coding behavior of one
important model in this class: autoencoders constructed with rectiﬁed linear hidden units. After
describing the intuitive behavior of these models, we examine their behavior on artiﬁcial and natural
datasets.
1
arXiv:1301.3753v2  [cs.LG]  19 Jan 2013
2 Background
One way to take advantage of a readily available unlabeled datasetOis to learn features that encode
the data well. Such features should then presumably be useful for describing the data effectively,
regardless of the particular task. Formally, we wish to ﬁnd a “dictionary” or “codebook” matrix
D ∈Rn×k whose columns can be combined or “decoded” linearly to represent elements from the
dataset. There are many ways to do this: for example, one could use random vectors or samples
from O, or deﬁne some cost function over the space of dictionaries and optimize to ﬁnd a good one.
Once Dis known, then for any input x, we would also like to compute the coefﬁcients h ∈Rk that
give the best linear approximation ˆx = Dh to x. This problem is often expressed as an optimization
of a squared error loss
h = arg min
u
∥Du −x∥2
2 + R(u)
where Ris some regularizer. If R= 0, for example, then this is linear regression; when R= ∥·∥1,
the encoding problem is known as sparse coding or lasso regression [14]. Sparse coding yields state-
of-the art results on many tasks for a wide variety of dictionary learning methods [2], but it requires a
full optimization procedure to compute h, which might require more or less computation depending
on the input. To address this complexity, Gregor and LeCun [6] tried approximating this coding
procedure by training a neural network (which has a bounded complexity) on a dataset labeled with
its sparse codes. For the remainder of the paper, we consider a similar problem—learning an efﬁcient
code for a set of data—but focus on simpler autoencoder neural network models to avoid solving
the complete sparse coding optimization problem.
2.1 Autoencoders
Autoencoders [8, 15, 10] are a versatile family of neural network models that can be used to learn
a dictionary from a set of unlabeled data, and to compute an encoding h for an input x. This
learning takes place simultaneously, since changing the dictionary changes the optimal encodings.
An autoencoder attempts to reconstruct its input after passing activation forward through one or
more nonlinear hidden layers. The general loss function for a one-layer autoencoder can be written
as
ℓ(O) = 1
2 |O|
∑
x∈O
∥ς(Dσ(Wx + b)) −x∥2
2 + R(O,D,W, b)
where ς(·) and σ(·) are activation functions for the output and hidden neurons, respectively,b ∈Rk
is a vector of hidden unit activation biases, and W ∈Rk×n is a matrix of weights for the hidden
units. By setting ς(z) =z, this family of models clearly belongs to the class of coding models, where
h = σ(Wx + b). This encoding scheme is easy to compute, but it also assumes that the optimal
coefﬁcients can be represented as a linear transform of the input, passed through some nonlinearity.
Commonly, the encoding and decoding weights are “tied” so that W = DT, which forces the
dictionary to take on values that are useful for both tasks. Seen another way, tied weights provide an
implicit orthonormality constraint on the dictionary [9]. Tying the weights also reduces by half the
number of parameters that the model needs to tune.
2.2 Rectiﬁed linear autoencoders
Traditionally, neural networks are constructed with sigmoid activations like σ(z) = (1 +ez)−1.
The rectiﬁed linear activation function [z]+ = max(0,z) has been shown to improve training and
performance in multilayer networks by addressing several shortcomings of sigmoid activation func-
tions [11, 5]. In particular, the rectiﬁed linear activation function produces a true 0 (not just a small
value) for negative input and then increases linearly. Unlike the sigmoid, which has a derivative
that vanishes for large input, the derivative of the rectiﬁed linear activation is either 0 for negative
inputs, or 1 for positive inputs This “switching” behavior is inspired by the ﬁring rate pattern seen
in some biological neurons, which are inactive for sub-threshold inputs and then increase ﬁring rate
as input magnitude increases. The true 0 output of the rectiﬁed linear activation function effectively
deactivates neurons that are not tuned for a speciﬁc input, isolating a smaller active model for that
input [5]. This also has the effect of combining exponentially many linear coding schemes into one
codebook [11].
2
xw
b
Hw,b
(a) Classiﬁer view.
xw
b
Hx;b (b) Encoder view.
Figure 1: Visualizing the half-spaces associated with data point x and feature vector w. In a linear
classiﬁer (left), weight vectors deﬁne hyperplanes that split data points into two groups. In a linear
encoder (right), a data point x deﬁnes a hyperplane which is shifted in the direction opposite that
data point through a distance b associated with feature w; the hidden unit corresponding to w is
coded by the distance from w to the shifted hyperplane.
To learn a dictionary from data, Glorot et al. [5] proposed the single-layer rectiﬁed linear autoen-
coder with tied weights by setting W = DT, ς(z) =zand σ(z) = [z]+ in the general autoencoder
loss, giving
ℓ(O) = 1
2 |O|
∑
x∈O
D
[
DTx + b
]
+ −x

2
2
+ R(O,D, b).
Hidden units in rectiﬁed linear autoencoders behave similarly to the “triangle K-means” encoding
proposed by Coates et al. [1], which produces a coefﬁcient vector h for input x such that hi =
[ µ−∥x −ci∥]+, where µ= 1
k
∑k
j=1 ∥x −cj∥is the mean distance to all cluster centroids. Like
a rectiﬁed linear model, triangle K-means produces true 0 activations for many (approximately half)
of the clusters, while coding the rest linearly. An equivalent [4] variation is the “soft thresholding”
scheme [2], where hi =
[
dT
i x −λ
]
+ for some parameter λ. This coding scheme is equivalent to a
rectiﬁed linear autoencoder with a shared bias λon all hidden units. Rectiﬁed linear autoencoders
generalize all of the models in this “switched linear” class and provide a uniﬁed form for their loss
functions.
2.3 Whitening
The switched linear family of models uses linear codes for some regions of input space and produces
zeros in others. These models perform best when applied to whitened datasets. In fact, Le et
al. [9] showed that sparse autoencoders, sparse coding, and ICA all compute the same family of
loss functions, but only if the input data are approximately white. Empirically, Coates et al. [1,
2] observed that for several different single-layer coding models, switched-linear coding schemes
perform best when combined with pre-whitened data.
Whitening seems to be important in biological systems as well: Although the human eye does not
receive white input, neurons in retina and LGN appear to whiten local patches of natural images
[13, 3] before further processing in the visual cortex.
3 Switched linear codes
Many researchers (e.g., [7, 11, 5]) have described how the rectiﬁed linear activation function sep-
arates its inputs into two natural groups of outputs: those with 0 values, and those with positive
values. We belabor this observation here to make a further point about the behavior of rectiﬁed
linear autoencoders below.
3
−10
−5
0
5
10 −10
−5
0
5
10
−10
−5
0
5
10
−10 −5 0 5 10
−10
−5
0
5
10
Figure 2: Rectiﬁed linear feature planes for a 3D gaussian dataset, using a complete dictionary
(k= 3). With an undercomplete or 1×complete dictionary, these planes tend to move to the bounds
of the dataset, forming a new coordinate system. The image on the left is a full 3D view of the space,
while the image on the right is a projection onto the x–yplane.
Consider a data point x and each encoding feature (row) wj of DT geometrically as vectors in Rn,
shown in Figure 1. In the context of a linear classiﬁer, the weight vector w is often described as
deﬁning the normal to a hyperplane Hw that divides the dataset into two subsets. A linear encoder,
on the other hand, is attempting to determine which hidden units describe its input, so in this context
it is the vectorx that deﬁnes the normal to a hyperplaneHx passing through the origin. Each feature
wj in the autoencoder corresponds to a bias term bj; this bias can be seen as translating Hx in the
direction away from x through a distance of bj. After this translation, if wj lies on the opposite
side of Hx;wj,bj from x, then the hidden unit hj will be set to 0 by a rectiﬁed activation function. 1
In other words, rectiﬁcation selectively deactivates features wj in the loss for x whenever feature
wj points sufﬁciently far away from x.2 After hidden unit hj is deactivated, decoding entry wj no
longer has any effect on the output. We can thus deﬁne
ψx = {j : wjx + bj >0}
as the set of features that have nonzero activations in the hidden layer of the autoencoder for input
x. Then
hψ = [hψ1 ,...,h ψr ]T
is the vector of hidden activations for just the features in ψ, Dψ is the matrix of corresponding
columns from D, and bψ are the corresponding bias terms. This gives a new loss function
ℓψ(O) = 1
2 |O|
∑
x∈O
Dψ
(
DT
ψx + bψ
)
−x
2
2 + R(O,D, b)
that is computed only over the active features ψx for each input x.
With b = 0 and a sparse regularizer R = ∥h∥1, the loss ℓψ reduces to that proposed by Le et al.
[9] for ICA with a reconstruction cost. For whitened input, then, the rectiﬁed linear autoencoder
behaves not only like a mixture of exponentially many linear models [11], but rather like a mixture
of exponentially many ICA models—one ICA model is selected for each input x, but all features are
selected from, and must coexist in, one large feature set. Importantly, this whitening requirement
only holds for the region of data delimited by the set of active features, potentially simplifying the
whitening transform.
Mirrored by the geometric interpretation, from the perspective of a data point, nonzero bias terms
can be thought of as translating the data to a new origin along the axis of that feature. Seen from
the perspective of a feature, the bias terms can help move the active region for that feature to an
area of space where data are available to describe—potentially ignoring other regions of space that
contain other portions of the data. From either viewpoint, the bias parameters in the model can
1Analytically, zj = wj x + bj < 0, so hj = max(0, zj ) = 0.
2Where "sufﬁciently" is determined by the bias for that feature.
4
−15 −10 −5 0 5 10 15 −15−10−5 0 51015
−15
−10
−5
0
5
10
15
−15 −10 −5 0 5 10 15
−15
−10
−5
0
5
10
15
Figure 3: Sigmoid feature planes for a 3D gaussian dataset, using an overcomplete dictionary ( k =
6). The weights corresponding to each hidden neuron form the normal vector to each plane, and the
distance from the origin is given by the bias for that unit. In the above example, three of the encoding
planes have partitioned the major axis of the data distribution (perhaps to maximize effective coding
regions), while the remaining planes capture variance along minor axes of the dataset. The image
on the left is a full 3D view of the space, while the image on the right is a projection onto the x–y
plane.
−10
−5
0
5
10
−10
−5
0
5
10
−10
−5
0
5
10
−10 −5 0 5 10
−10
−5
0
5
10
Figure 4: Rectiﬁed linear feature planes for a 3D gaussian dataset, using an overcomplete dictionary
(k= 6). These feature planes group together in pairs, splitting the dataset along one axis. The image
on the left is a full 3D view of the space, while the image on the right is a projection onto the x–y
plane.
thus help compensate for non-centered data. Figure 2 shows typical behavior for a 1×complete
dictionary of rectiﬁed linear units when trained on an artiﬁcial 3D gaussian dataset. In this case,
each feature plane uses its associated bias to translate away from the origin, together providing a
sort of “bounding box” for the data. Though the axes of the box are not necessarily orthogonal, the
features deﬁne a new basis that is often aligned with or near the principal axes of the data.
3.1 Effective coding region
Visualizing feature hyperplanes is actually instructive for many types of activation functions. Fig-
ure 3, for instance, shows the feature planes for a 2×overcomplete tied-weights autoencoder with
sigmoid hidden units, trained on an artiﬁcial 3D gaussian dataset. 3 Several of the features learned
by this model partition the data along the principal axis of variance, while the remaining features
capture some of the variance along minor axes of the dataset. In comparison, a 2×overcomplete
rectiﬁed linear autoencoder trained on a 3D gaussian dataset (Figure 4) creates pairs of negated fea-
ture vectors, in effect providing a full-wave rectiﬁed linear output along each axis. Together, these
3The feature planes in this case are aligned with the logistic activation value 0.5.
5
2
 1
 0 1 22.0
1.5
1.0
0.5
0.0
0.5
1.0
1.5
2.0
Figure 5: Rectiﬁed linear responses summed over input space for mixture of gaussian datasets. The
plot on the left was trained with no regularization, while the plots in the middle and on the right
were trained with an L1 sparsity penalty on the hidden activations. Color delineates regions of equal
potential summed over all hidden units, and small red circles indicate the learned feature vectors.
Data points are shown in black.
pairs of planes split the data into orthants, which produces a coding scheme that is 50 % sparse
because only half of the features are active for any given region of the input space.
Clearly these two activation functions have different ways of modeling the input data, but why do
these particular patterns of organization appear? In our view, the unbounded linear output of the
rectiﬁed linear autoencoder units addresses a subtle but important issue inherent in coding with
sigmoid activation functions: scale. Consider, for example, a two-dimensional dataset consisting of
points along a one-dimensional manifold
L=
{
[x,ϵ]T : 0<x<S
}
where ϵ is gaussian noise. Regardless of the activation function, this network requires just one
hidden unit to represent points in L: we can set the bias to 0 and weights to [1,0]T to align the
feature of the hidden unit with the linear function that describes the data. Then points in the dataset
are simply coded by their distance from the origin, as given by the output of the activation function.
For a sigmoid activation function, however, asSincreases, the activation function saturates, limiting
the power of the hidden units to discriminate between small changes in xnear S. This saturation
limits the scale at which sigmoid units can describe data effectively, but it also prevents numerical
instability by limiting the range of the output.
Regardless of the activation function, then, each feature vector can be seen as normal to a hyperplane
in input space. The activation function encodes data along the axis of the feature as a function of
the distance to the hyperplane for that feature. Sigmoid activation functions saturate for points far
from the hyperplane, whereas linear activations solve this “vanishing signal” problem by coding all
x values with a linear output response. Rectiﬁed linear activations make a further distinction by
coding only one half of the input space. However, the outputs of a linear activation are unbounded,
so networks with inputs at large scales are prone to numerical instability during training.
4 Behavior on complex data
So far, the visualizations tools that we have used are based on simple gaussian distributions of
data. Many interesting datasets, however, are emphatically not gaussian—indeed, one of the primary
reasons that ICA is so effective is that it explicitly searches for a non-gaussian model of the data!
In this section we present several visualizations for these models using more complex datasets: ﬁrst
we examine mixtures of gaussians, and then we use the MNIST digits dataset as a foray into a more
natural set of data.
We ﬁrst tested the behavior of rectiﬁed linear autoencoders on a small, 2D mixture of gaussians
dataset. (See Figure 5.) After training, even massively overcomplete dictionaries tended to display
the feature pairing behavior described above, particularly when combined with a sparse regularizer,
suggesting that these networks are capable of identifying the number of features required to code
the data effectively. Unfortunately, while it provides simple visualization, working in 2D does not
necessarily generalize to higher dimensions, as our intuition in low-dimensional spaces can quickly
lead us astray in larger spaces.
6
Figure 6: Summary plots of individual feature response regions for a rectiﬁed linear autoencoder
trained on a mixture of gaussians dataset, including an L1 sparsity penalty. Colored bands indicate
equipotential regions of response for the feature, and data points are shown in black. Interestingly,
most of the features are inactive for this dataset, with only three features devoted to coding important
regions of the data.
Figure 7: PCA features (eigendigits) computed for MNIST digits.
4.1 MNIST digits
The MNIST dataset consists of 60000 28 ×28 grayscale images of the handwritten digits 0 through
9. Figure 7 shows the PCA “eigendigits” computed using this dataset; these digits point in the
directions of highest variance for this dataset overall, and are often interpreted as a coding of the digit
using a Fourier basis. Figure 8 shows the features with the highest bias values computed by a single-
layer rectiﬁed linear autoencoder on the same dataset. For visualization purposes, each feature w
on the left of each column is paired with its maximally negative feature v = arg minu wTu from
the dictionary. Many of the primary features in the dictionary are negative images of each other,
indicating that the “pairing” of feature planes observed for low-dimensional datasets also occurs
with higher-dimensional data.
Finally, Figure 9 shows features learned by a two-layer rectiﬁed linear autoencoder with untied
weights. This model was trained with just 64 ﬁrst-layer hidden units, and 1024 second-layer units.
As observed in the low-dimensional gaussian data, the features from the ﬁrst layer appear to model
the principal axes of the data, providing a bounding box of 64 dimensions that encodes the data for
the next layer. The features from the second layer resemble a more traditional sparse code, consisting
in this case of small segments of pen strokes. It is important to note that the training dataset was not
7
Figure 8: Some of the features computed using a single-layer rectiﬁed linear autoencoder with
a 1.5×overcomplete dictionary. Each feature w is displayed on the left, and on the right is its
corresponding maximally negative feature v = arg minu wTu.
(a) 64 ﬁrst-layer features.
(b) 1024 second-layer features.
Figure 9: MNIST features computed using a two-layer rectiﬁed linear autoencoder.
whitened beforehand, and no regularization was used to train the network—the autoencoder learned
these features automatically, using only a squared error reconstruction cost.
5 Conclusion
This paper has synthesized many recent developments in encoding and neural networks, with a focus
on rectiﬁed linear activation functions and whitening. It also presented an intuitive interpretation for
the behavior of these encodings through simple visualizations. Sample features learned from both
artiﬁcial and natural datasets provided examples of learning behavior when these algorithms are
applied to different types of data.
There are many more paths to explore in this area of machine learning research. Whitening has
appeared in many places throughout this paper and seems to be a very important component of linear
autoencoders and coding systems in general. However, this connection seems poorly understood,
and would beneﬁt from further exploration, particularly with respect to the idea of whitening local
regions of the input space.
8
References
[1] A. Coates, H. Lee, and A.Y . Ng. An analysis of single-layer networks in unsupervised feature
learning. In AISTATS, volume 1001, page 48109, 2011.
[2] A. Coates and A.Y . Ng. The importance of encoding versus training with sparse coding and
vector quantization. In International Conference on Machine Learning , volume 8, page 10,
2011.
[3] Y . Dan, J.J. Atick, and R.C. Reid. Efﬁcient coding of natural scenes in the lateral genic-
ulate nucleus: experimental test of a computational theory. The Journal of Neuroscience ,
16(10):3351–3362, 1996.
[4] M. Denil and N. de Freitas. Recklessly approximate sparse coding, 2012.
[5] X. Glorot, A. Bordes, and Y . Bengio. Deep sparse rectiﬁer neural networks. In Proceedings
of the Fourteenth International Conference on Artiﬁcial Intelligence and Statistics (AISTATS
2011), 2011.
[6] K Gregor and Y LeCun. Learning fast approximations of sparse coding. In Proc. International
Conference on Machine learning (ICML’10), 2010.
[7] R.H.R. Hahnloser, H.S. Seung, and J.J. Slotine. Permitted and forbidden sets in symmetric
threshold-linear networks. Neural Computation, 15(3):621–638, 2003.
[8] G.E. Hinton and R.S. Zemel. Autoencoders, minimum description length, and Helmholtz free
energy. Advances in Neural Information Processing Systems, pages 3–3, 1994.
[9] Q.V . Le, A. Karpenko, J. Ngiam, and A.Y . Ng. ICA with reconstruction cost for efﬁcient
overcomplete feature learning. In Advances in Neural Information Processing Systems, 2011.
[10] A. Lemme, R.F. Reinhart, and J.J. Steil. Efﬁcient online learning of a non-negative sparse
autoencoder. In Proceedings of the ESANN, pages 1–6, 2010.
[11] V . Nair and G.E. Hinton. Rectiﬁed linear units improve restricted boltzmann machines. In
27th International Conference on Machine Learning , pages 807–814. Omnipress Madison,
WI, 2010.
[12] B.A. Olshausen and D.J. Field. Emergence of simple-cell receptive ﬁeld properties by learning
a sparse code for natural images. Nature, 381(6583):607–609, 1996.
[13] E.P. Simoncelli and B.A. Olshausen. Natural image statistics and neural representation.Annual
review of neuroscience, 24(1):1193–1216, 2001.
[14] R. Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical
Society. Series B (Methodological), pages 267–288, 1996.
[15] P. Vincent, H. Larochelle, Y . Bengio, and P.A. Manzagol. Extracting and composing robust
features with denoising autoencoders. In Proceedings of the 25th international conference on
Machine learning, pages 1096–1103. ACM, 2008.
9


Peer review:
[Review]:
In the deep learning community there has been a recent trend in 
moving away from the traditional sigmoid/tanh activation function to 
inject non-linearity into the model. One activation function that has 
been shown to work well in a number of cases is called Rectified 
Linear Unit (ReLU). 
Building on the prior research, this paper aims to provide an 
analysis of what is going on while training networks using these 
activation functions, and why do they work. In particular the authors 
provide their analysis from the context of training a linear auto-encoder 
with rectified linear units on a whitened data. They use a toy dataset in 
3 dimensions (gaussian and mixture of gaussian) to conduct the analysis. 
They loosely test the hypothesis obtained from the toy datasets on the 
MNIST data.

Though the paper starts with a lot of promise, unfortunately it fails to 
deliver on what was promised. There is nothing in the paper (no new 
idea or insight) that is either not already known, or fairly straightforward 
to see in the case of linear auto-encoders trained using a rectified 
linear thresholding unit. Furthermore there are a number of flaws in 
the paper. For instance, the analysis of section 3.1 seems to be a bit 
mis-leading. By definition if one fixes the weight vector w to [1,0] there 
is no way that the sigmoid can distinguish between x's which are 
greater than S for some S. However with the weight vector taking 
arbitrary continuous values, that may not be the case. Besides, the 
purpose of the encoder is to learn a representation, which can best 
represent the input, and coupled with the decoder can reconstruct it. 
The encoder learning an identity function (as is argued in the paper) is not 
of much use. Finally, the whole analysis of section 3 was based on a 
linear auto-encoder, whose encoder-decoder weights were tied. However 
in the case of MNIST the authors show the filters learnt from an untied 
weight auto-encoder. There seems to be some disconnect there. 

In short the paper does not offer any novel insight or idea with respect 
 to learning representation using auto-encoders with rectified linear 
thresholding function. Various gaps in the analysis also makes it a not 
very high quality work.