Learning Invariant Representations with Local Transformations

Kihyuk Sohn
kihyuks@umich.edu
Honglak Lee
honglak@eecs.umich.edu
Dept. of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, MI 48109, USA

Abstract
Learning invariant representations is an important problem in machine learning andpattern recognition.
In this paper, we
present a novel framework of transformationinvariant feature learning by incorporating linear transformations into the feature learning algorithms.
For example,
we present the transformation-invariant restricted Boltzmann machine that compactly
represents data by its weights and theirtransformations, which achieves invarianceof the feature representation via probabilistic max pooling. In addition, we show that
our transformation-invariant feature learning
framework can also be extended to other unsupervised learning methods, such as autoencoders or sparse coding. We evaluate ourmethod on several image classification benchmark datasets, such as MNIST variations,
CIFAR-10, and STL-10, and show competitive or superior classification performance
when compared to the state-of-the-art. Furthermore, our method achieves state-of-theart performance on phone classification tasks
with the TIMIT dataset, which demonstrates
wide applicability of our proposed algorithms
to other domains.

1. Introduction
In recent years, unsupervised feature learning algorithms have emerged as promising tools for learning
representations from data (Hinton et al., 2006; Bengio
et al., 2007; Ranzato et al., 2006). In particular, it
is an important problem to learn invariant representations that are robust to variability in high-dimensional
data (e.g., images, speech, etc.) since they will enAppearing in Proceedings of the 29 th International Conference on Machine Learning, Edinburgh, Scotland, UK, 2012.
Copyright 2012 by the author(s)/owner(s).

able machine learning systems to achieve good generalization performance while using a small numberof labeled training examples. In this context, several feature learning algorithms have been proposed
to learn invariant representations for specific transformations by using customized approaches. For example, convolutional feature learning methods (Lee et al.,2011; Kavukcuoglu et al., 2010; Zeiler et al., 2010) canachieve shift-invariance by exploiting convolution operators. As another example, the denoising autoencoder (Vincent et al., 2008) can learn features thatare robust to the input noise by trying to reconstruct
the original data from the hidden representation of theperturbed data. However, learning invariant representations with respect to general types of transformations is still a challenging problem.
In this paper, we present a novel framework oftransformation-invariant feature learning. We focus
on local transformations (e.g., small amounts of translation, rotation, and scaling in images), which canbe approximated as linear transformations, and incorporate linear transformation operators into the feature learning algorithms. For example, we present thetransformation-invariant restricted Boltzmann machine, which is a generative model that represents input data as a combination of transformed weights.
In this case, a transformation-invariant feature representation is obtained via probabilistic max poolingof the hidden units over the set of transformations. In
addition, we show extensions of our transformationinvariant feature learning framework to other unsupervised feature learning algorithms, such as autoencoders or sparse coding.
In our experiments, we evaluate our method on thevariations of the MNIST dataset and show that ouralgorithm can significantly outperform the baseline restricted Boltzmann machine when underlying transformations in the data are well-matched to those considered in the model. Furthermore, our method canlearn features that are much more robust to the wide
range of local transformations, which results in highlyLearning Invariant Representations with Local Transformations

competitive performance in visual recognition tasks
on CIFAR-10 (Krizhevsky, 2009) and STL-10 (Coates
et al., 2011) datasets. In addition, our method alsoachieves state-of-the-art performance on phone classification tasks with the TIMIT dataset, which demonstrates wide applicability of our proposed algorithms
to other domains.

via stochastic gradient descent. Although computing
the exact gradient is intractable, we can approximate
it using contrastive divergence (Hinton, 2002).

The rest of the paper is organized as follows. We provide the preliminaries in Section 2, and in Section 3, we
introduce our proposed transformation-invariant feature learning algorithms. In Section 4, we review theprevious work on invariant feature learning. Then, in
Section 5, we report the experimental results on several datasets. Section 6 concludes the paper.

In this section, we formulate a novel feature learning
framework that can learn invariance to a set of linear transformations based on the RBM. We begin thesection with describing the transformation operator.

2. Preliminaries
In this paper, we present a general framework forlearning locally-invariant features using transformations. For presentation, we will use the restricted
Boltzmann machine (RBM) as the main example.1 We
first describe the RBM below, followed by its novel extension (Section 3).
The restricted Boltzmann machine is a bipartite undirected graphical model that is composed of visible andhidden layers. Assuming binary-valued visible andhidden units, the energy function and the joint probability distribution are given as follows:2E(v, h)=P (v, h)=−vT Wh − bT h − cT v,
1
exp (−E(v, h))
Z

D

(1)
(2)
K

where v ∈ {0, 1} are binary visible units, h ∈ {0, 1}
are binary hidden units, and W ∈ RD×K , b ∈ RK ,and c ∈ RD are weights, hidden biases, and visible biases, respectively. Z is a normalization factor that depends on the parameters {W, b, c}. Since RBMs haveno intra-layer connectivity, exact inference is tractableand block Gibbs sampling can be done efficiently using
the following conditional probabilities:Xp(hj = 1|v) = sigmoid(vi Wij + bj )(3)iXp(vi = 1|h) = sigmoid(
hj Wij + ci )(4)jwhere sigmoid(x) = 1+exp1 (−x) . We train the RBM
parameters by minimizing the negative log-likelihood
1
We will describe extensions to other feature learningalgorithms in Section 3.4.
2
Due to space constraints, we present only the caseof binary-valued input variables; however, the RBM withreal-valued input variables can be formulated straightforwardly (Hinton & Salakhutdinov, 2006; Lee et al., 2008).

3. Learning Transformation-Invariant
Feature Representations
3.1. Transformation-invariant RBM

The transformation operator is defined as a mapping
T : RD1 → RD2 that maps D1 -dimensional input vectors into D2 -dimensional output vectors (D1 ≥ D2 ).
In our case, we assume a linear transformation matrix
T ∈ RD2 ×D1 , i.e., each coordinate of the output vector is represented as a linear combination of the input
coordinates.
With this notation, we formulate the transformationinvariant restricted Boltzmann machine (TIRBM)
that can learn invariance to a set of transformations.
Specifically, for a given set of transformation matrices
Ts (s = 1, · · · , S), the energy function of TIRBM is
defined as follows:E(v, H) =
−

K X
S
X
j=1 s=1

s.t.

S
X(Ts v)T wj hj,s −

(5)
K X
S
X

bj,s hj,s − cT v

j=1 s=1

hj,s ≤ 1, hj,s ∈ {0, 1}, j = 1, · · · , K,

(6)

s=1

where v are D1 -dimensional visible units, and wj are
D2 -dimensional (filter) weights corresponding to thej-th hidden unit. The hidden units are represented asa matrix H ∈ {0, 1}K×S with hj,s as its (j, s)-th entry.
PS
In addition, we denote zj = s=1 hj,s , zj ∈ {0, 1} as apooled hidden unit over the transformations.
In Equation (6), we impose a softmax constraint on
hidden units so that at most one unit is activated ateach row of H. This probabilistic max pooling3 allows us to obtain a feature representation invariant
to linear transformations. More precisely, supposethat the input v1 matches the filter wj . Given another input v2 that is a transformed version of v1 , the
TIRBM will try to find a transformation matrix Tsj sothat the v2 matches the transformed filter TsTj wj , i.e.,
v1T wj ≈ v2T TsTj wj .4 Therefore, v1 and v2 will both3
A similar technique is used in convolutional deep belief
networks (Lee et al., 2011), in which spatial probabilistic
max pooling is applied over a small spatial region.
4
Note that the transpose TsT of a transformation ma-

Learning Invariant Representations with Local Transformations
v1
wi

TsiTwiTsjTwiv2
wj

zi (v1)=1

TsiTwjTsjTwjwi

TsiTwiTsjTwi

TsiTwjwj(hi,si=1)zi (v2)=1

zj (v1)=1

Input
TsjTwj(hj,sj =1)
zj (v2)=1(transformed)
filters
Probabilistic
max pooling

Pooled features

Figure 1. Feature encoding of TIRBM. Here, v1 and v2 denote two image patches, and the shaded pattern inside v2
reflects v1 . The shaded patterns in transformed filters show the corresponding original filters wi or wj . The filters
selected via probabilistic max pooling across the set of transformations are shown in red arrows (e.g., in the rightmost
example, the hidden unit hj,sj corresponding to the transformation Tsj and the filter wj contributes to activate zj (v2 ).)In this illustration, we assumed D1 = D2 and also the existence of the identity transformation.

activate zj after probabilistic max pooling. Figure 1
illustrates this idea.
Compared to the regular RBM, the TIRBM can learn
more diverse patterns, while keeping the number of parameters small. Specifically, multiplying transformation matrix (e.g., TsT wj ) can be viewed as increasing
the number of filters by the factor of S, but without
significantly increasing the number of parameters due
to parameter sharing. In addition, by pooling over
local transformations, the filters can learn invariant
representations (i.e., zj ’s) to these transformations.
The conditional probabilities are computed as follows:exp (wjT Ts v + bj,s )P
1 + s0 exp (wjT Ts0 v + bj,s0 )Xp(vi = 1|h) = sigmoid( (TsT wj )i hj,s + ci )p(hj,s = 1|v) =

(7)
(8)

j,s

Similar to RBM training, we use stochastic gradientdescent to train TIRBM. The gradient of the loglikelihood is approximated via contrastive divergenceby taking the gradient of the energy function (Equation (5)) with respect to the model parameters.
3.2. Sparse TIRBM
The sparseness of the feature representation is often
a desirable property. By following Lee et al.’s (2008)
approach, we can extend our model to sparse TIRBM
by adding the following regularizer for a given set ofdata {v(1) , · · · , v(N ) } to the negative log-likelihood:
!
N
K
X
1 X
(n) (n)E[zj |v ]
(9)
Lsp =
D p,
N n=1
j=1

The expectation of pooled activation is written asPTs exp (wj Ts v + bj,s )PE[zj |v] =.
(10)
1 + s exp (wjT Ts v + bj,s )Note that we regularize over the pooled hidden units
zj rather than individual hidden units hj,s . In our
experiments, we used L2 distance for D(·, ·), but one
can also use KL divergence for the sparsity penalty.
3.3. Generating transformation matrices
In this section, we discuss how to design the transformation matrix T . For the ease of presentation, we
assume 1-d transformations, but it can be extended
to 2-d cases (e.g., image transformations) straightforwardly. Further, we assume the case of D1 = D2 = D
here; but, we will discuss general cases later.
As mentioned previously, T ∈ RD×D is a linear transformation matrix from x ∈ RD to y ∈ RD ; i.e., eachcoordinate of y is constructed via linear combinationof the coordinates in x with weight matrix T as follows:
yi =

D
X

Tij xj , ∀i = 1, · · · , D.

(11)

j=1

For example, shifting by s can be defined as(
1 if i = j + s,
Tij (s) =0 otherwise.
For 2-d image transformations such as rotation andscaling, the contribution of input coordinates to each
output coordinate is computed with bilinear interpolation. Since T ’s are pre-computed once and usuallysparse, Equation (11) can be computed efficiently.
3.4. Extensions to other methods

where D is a distance function; p is the target sparsity.
trix Ts also induces a linear transformation. In the paper,
we will occasionally abuse the term “transformation” todenote these two cases, as long as the context is clear.

We emphasize that our transformation-invariant feature learning framework is not limited to the energybased probabilistic models, but can be extended toother unsupervised learning methods as well.

Learning Invariant Representations with Local TransformationsFirst, it can be readily adapted to autoencoders bydefining the following softmax encoding and sigmoid
decoding functions:exp (wjT Ts v + bj,s )P
1 + s0 exp (wjT Ts0 v + bj,s0 )Xv̂i = sigmoid( (TsT wj )i fj,s + ci )fj,s (v) =

(12)
(13)

j,s

Following the idea of TIRBM, we can also formulate
the transformation-invariant sparse coding as follows:
min

W,H

K X
S
X X
(n)
k
TsT wj hj,s − v(n) k2 ,n

(14)

j=1 s=1

s.t. kHk0 ≤ γ, kH(j, :)k0 ≤ 1, kwj k2 ≤ 1,

(15)

where γ is a constant. The second constraint in (15)can be understood as an analogy to the softmax constraint in Equation (6) of TIRBMs.
Similar to standard sparse coding, we can optimize
the parameters by alternately optimizing W and H
while fixing the other. Specifically, H can be (approximately) solved using Orthogonal Matching Pursuit,and therefore we refer this algorithm a transformationinvariant Orthogonal Matching Pursuit (TIOMP).

4. Related Work
Researchers have made significant efforts to develop invariant feature representations. For example, the rotation- or scale-invariant descriptors, such
as SIFT (Lowe, 1999), have shown a great success inmany computer vision applications. However, theseimage descriptors usually demand a domain-specificknowledge with a significant amount of hand-crafting.
As an alternative approach, several unsupervised
learning algorithms have been proposed to learn robustfeature representations automatically from the sensory
data. As an example, the denoising autoencoder (Vincent et al., 2008) can learn robust features by trying
to reconstruct the original data from the hidden representations of randomly perturbed data generated fromthe distortion processes, such as adding noise or multiplying zeros for randomly selected coordinates.
Among types of transformations relating to temporallyor spatially correlated data, translation has been extensively studied in the context of unsupervised learning. Specifically, convolutional training (LeCun et al.,
1989; Kavukcuoglu et al., 2010; Zeiler et al., 2010;
Lee et al., 2011; Sohn et al., 2011) is one of the mostpopular methods that encourages shift-invariance during the feature learning. For example, the convolutional deep belief network (CDBN) (Lee et al., 2011),

which is composed of multiple layers of convolutional
restricted Boltzmann machines and probabilistic max
pooling, can learn a representation invariant to local
translation.
Besides translation, however, learning invariant features for other types of image transformations havenot been extensively studied. In contemporary workof ours, Kivinen and Williams (2011) proposed thetransformation equivariant Boltzmann machine, whichshares a similar mathematical formulation to our models in that both try to infer the best matching filters by transforming them using linear transformation
matrices. However, while their model was motivatedfrom the “global equivariance”, the main purpose of
our work is to learn locally-invariant features that canbe useful in classification tasks. Thus, rather thanconsidering an algebraic group of transformation matrices (e.g., “full rotations”), we focus on the varietyof local transformations that include rotation, translation as well as scale variations. Furthermore, weeffectively address the boundary effects that can behighly problematic in scaling and translation operators
by forming a non-square T matrix, rather than zeropadding.5 In addition, we present a general frameworkof transformation-invariant feature learning and show
extensions based on the autoencoder and sparse coding. Overall, our argument is strongly supported by
the state-of-the-art performance in image and audio
classification tasks.
As another related work, feature learning methods
with topographic maps can also learn invariant representations (Hyvärinen et al., 2001; Kavukcuoglu et al.,2009). Compared to these methods, our model is morecompact and has fewer parameters to train since it factors out the (filter) weights and their transformations.
In addition, given the same number of parameters, ourmodel can represent more diverse and variable input
patterns than topographic filter maps.

5. Experiments
We begin by describing the notation. For images, we
assume a receptive field size of r × r pixels (for input
image patches) and a filter size of w × w pixels. We
define gs to denote the number of pixels corresponding
to the transformation (e.g., translation or scaling). For
example, we translate the w × w filter across the r × r
receptive field with a stride of gs pixels (Figure 2(a)),
or scale down from (r−l·gs)×(r−l·gs) to w×w (where0 ≤ l ≤ b(r − w)/gsc) by sharing the same center for5

We observed that learning with zero-padded squaredtransformation matrices showed significant boundary effect in its visualization of filters, and this often resulted insignificantly worse classification performance.

Learning Invariant Representations with Local Transformations

rw

r
gs

(a) translation

Table 1. Test classification error on MNIST transformation
datasets. The best-performing methods for each dataset
are shown in bold.

gsw(b) scaling

Figure 2. Translation and scale transformations on images.

the filter and the receptive field (Figure 2(b)).
For classification tasks, we used the posterior probability of the pooled hidden unit (Equation (10)) as a feature. Note that the dimension of the extracted feature
vector for each image patch is K, not K × S. Thus,
we argue that the performance gain of the TIRBM
over the regular RBM comes from the better representation (i.e., transformation-invariant features), rather
than from the classifier’s use of higher-dimensional features.

Datasetrotrot-bgimg
scale
scale-bgrandtranstrans-bgrand

RBM
15.6%
54.0%
6.1%
32.1%
15.3%
57.3%

(a) basic(b) rot

TIRBM
4.2%
35.5%
5.5%
23.7%
9.1%
43.7%

Transformation
rotationrotation
scaling
scalingtranslationtranslation(c) scale(d) trans

5.1. Handwritten digit recognition with priortransformation information
First, we verified the performance of our algorithm on
the variations of a handwritten digit dataset, assuming that the transformation information is given. From
the MNIST variation datasets (Larochelle et al., 2007),
we tested on “mnist-rot” (rotated digits, referred to asrot) and “mnist-rot-back-image” (rotated digits withbackground images, referred to as rot-bgimg). To further evaluate with different types of transformations,
we created four additional datasets that contain scale
and translation variations with and without random
background (referred to as scale, scale-bgrand, trans,
and trans-bgrand, respectively).6 Some examples areshown in Figure 3.
For these datasets, we trained sparse TIRBMs on
the image patches of size 28 × 28 pixels with dataspecific transformations. For example, we considered
16 equally-spaced rotations (i.e., the step size of π8 )for the rot and rot-bgimg datasets. Similarly, for the
scale and scale-bgrand datasets, we generated scaletransformation matrices with w = 20 and gs = 2,
which can map from (28 − 2l) × (28 − 2l) pixels to
20 × 20 pixels with l ∈ {0, ..., 4}. For the trans andtrans-bgrand datasets, we set w = 24 and gs = 2
to have total nine translation matrices that cover the6

We followed the generation process described in
http://www.iro.umontreal.ca/~lisa/twiki/bin/view.
cgi/Public/MnistVariations to create the customized
scaled and translated digits. For example, we randomly
selected the scale-level uniformly from 0.3 to 1 or thenumber of pixel shifts in horizontal and vertical directions
without making any truncation of the foreground pixels.
For the datasets with random background, we randomly
added the uniform noise in [0, 1] to the background pixels.

(e) TIRBM on mnist-rot(f) RBM on mnist-rot

Figure 3. (top) Samples from the handwritten digit
datasets with (a) no transformations, (b) rotation, (c) scaling, and (d) translation. (bottom) Learned filters frommnist-rot dataset with (e) the sparse TIRBM and (f) thesparse RBM, respectively.

28 × 28 regions using 24 × 24 pixels with a horizontal
and vertical stride of 2 pixels. For classification, we
trained 1, 000 filters for both sparse RBMs and sparse
TIRBMs and used a softmax classifier. We used 10,000
examples for the training set, 2,000 examples for thevalidation set, and 50,000 examples for the test set.
As reported in Table 1, our method (sparse TIRBMs)
consistently outperformed the baseline method (sparseRBMs) for all datasets. These results suggest that the
TIRBMs can learn better representations for the foreground objects by transforming the filters. It is worth
noting that our error rates for the mnist-rot and mnistrot-back-image datasets are also significantly lower
than the best published results obtained with stacked
denoising autoencoders (Vincent et al., 2010) (9.53%
and 43.75%, respectively).
For qualitative evaluation, we visualize the learned filters on the mnist-rot dataset trained with the sparse
TIRBM (Figure 3(e)) and the sparse RBM (Figure 3(f)), respectively. The filters learned from sparse
TIRBMs show much clearer pen-strokes than thoselearned from sparse RBMs, which partially explains

Learning Invariant Representations with Local Transformations(a) sparse RBM (baseline)(b) sparse TIRBM with translation(c) sparse TIRBM with rotation(d) sparse TIRBM with scaling
Figure 4. Visualization of filters trained with RBM and
TIRBMs on natural images. We trained 24 filters and used
nine translations with a step size of 1 pixel, five rotations
with a step size of π/8 radian, and two-level scale transformations with a step size (gs) of 2 pixels, respectively.

the impressive classification performance.
5.2. Learning invariant features from natural
imagesFor handwritten digit recognition in the previous section, we assumed prior information on global transformations on the image (e.g., translation, rotation,and scale variations) for each dataset. This assumption enabled the proposed TIRBMs to achieve significantly better classification performance than the baseline method, since the data-specific transformation information was encoded in the TIRBM.
However, for natural images, it is not reasonable to assume such global transformations due to the complex
image structures. In fact, recent literature (Yu et al.,2011; Lee et al., 2011; Vincent et al., 2008) suggeststhat some level of invariance to local transformations(e.g., few pixel translation or coordinate-wise noise)leads to improved performance in classification. From
this viewpoint, it makes more sense to learn representations with local receptive fields that are invariant
to generic image transformations (e.g., small amountsof translation, rotation, and scaling), which does not
require data-specific prior information.
We visualize the learned TIRBM filters in Figure 4,
where we used the 14×14 natural image patches takenfrom the van Hateren dataset (van Hateren & van der
Schaaf, 1998). The baseline model (sparse RBM)
learns many similar vertical edges (Figure 4(a)) thatare shifted by a few pixels, whereas our methods canlearn diverse patterns, including diagonal and horizontal edges, as shown in Figure 4(b), 4(c), and 4(d).

Table 2. Test classification accuracy on CIFAR-10 dataset.
1,600 filters were used unless otherwise stated. The numbers with † and ‡ are from (Coates et al., 2011) and (Coates& Ng, 2011a), respectively.
Methods (Linear SVM)
sparse RBM† (baseline)
sparse TIRBM (scaling only)
sparse TIRBM (rotation only)
sparse TIRBM (translation only)
sparse TIRBM (combined)
sparse TIRBM (combined, K= 4,000)
TIOMP-1/T (combined)
TIOMP-1/T (combined, K= 4,000)
VQ (K= 4,000)†
OMP-1/T (K= 1,600)‡
OMP-1/T (K= 6,000)‡
convolutional DBN (Krizhevsky, 2010)
deep NN (Ciresan et al., 2011)
deep NN (Coates & Ng, 2011b)

Accuracy
72.4%
76.8%
77.7%
77.5%
78.8%
80.1%
80.7%
82.2%
79.6%
79.4%
81.5%
78.9%
80.5%
82.0%

Table 3. Test classification accuracy on STL-10. 1,600 filters were used for all experiments.
Methods (Linear SVM)
sparse RBM
sparse TIRBM (scaling only)
sparse TIRBM (rotation only)
sparse TIRBM (translation only)
sparse TIRBM (combined)
VQ (K= 1,600) (Coates & Ng, 2011a)
SC (K= 1,600) (Coates & Ng, 2011a)
deep NN (Coates & Ng, 2011b)

Acc.± std.
55.0 ± 0.5%
55.9 ± 0.7%
57.0 ± 0.7%
57.8 ± 0.5%
58.7 ± 0.5%
54.9 ± 0.4%
59.0 ± 0.8%
60.1 ± 1.0%

These results suggest that TIRBMs can learn diverse
sets of filters, which is reminiscent of the effects of convolutional training (Kavukcuoglu et al., 2010). However, our model is much easier to train than convolutional models, and it can further handle generic transformations beyond translations.
5.3. Object recognition
We evaluated on image classification tasks using two
datasets. First, we tested on the widely used CIFAR10 dataset (Krizhevsky, 2009), which is composed of
50,000 training and 10,000 testing examples with 10
categories. Rather than learning features from thewhole image (32 × 32 pixels), we trained TIRBMs on
local image patches while keeping the RGB channels.
As suggested by Coates et al.(2011), we used the fixed
filter size w = 6 and determined the receptive field size
depending on the types of transformations.7 Then, after unsupervised training with TIRBM, we used theconvolutional feature extraction scheme, also following the Coates et al.(2011). Specifically, we computed
the TIRBM pooling-unit activations for each local r×r
7
For example, we used r = 6 for rotations. For bothscale variations or translations, we used r = 8 and gs = 2.

Learning Invariant Representations with Local Transformations

pixel patch that was densely extracted with a stride of
1 pixel, and averaged the patch-level activations overeach of the 4 quadrants in the image. Eventually, thisprocedure yielded 4K-dimensional feature vectors foreach image, which were fed into an L2-regularized linear SVM. We performed 5-fold cross validation to determine the hyperparameter C.
For comparison to the baseline model, we separately
evaluated the sparse TIRBMs with a single type oftransformation (translation, rotation, or scaling) using K = 1, 600. As shown in Table 2, each single
type of transformation in TIRBMs brought a significant performance gain over the baseline sparse RBMs.
The classification performance was further improvedby combining different types of transformations into asingle model.
In addition, we also report the classification results
obtained using TIOMP-1 (see Section 3.4) for unsupervised training. In this experiment, we used the following two-sided soft thresholding encoding function:
fj = max max(wjT Ts v − α, 0)sfj+K = max max(−wjT Ts v − α, 0) ,s

where α is a constant threshold that was crossvalidated. As a result, we observed about 1% improvement over the baseline method (OMP-1/T) using
1,600 filters, which supports the argument that ourtransformation-invariant feature learning framework
can be effectively transferred to other unsupervised
learning methods. Finally, by increasing the number offilters (K= 4,000), we obtained better results (82.2%)
than the previously published results using single-layer
models, as well as those using deep networks.
We also performed the object classification task on
STL-10 dataset (Coates et al., 2011), which is morechallenging due to the smaller number of labeled training examples (100 per class for each training fold).
Since the original images are 96×96 pixels, we downsampled the images into 32×32 pixels, while keeping
the RGB channels. We followed the same unsupervised training and classification pipeline as we did for
CIFAR-10. As reported in Table 3, there were consistent improvements in classification accuracy by incorporating the various transformations in learning algorithms. Finally, we achieved 58.7% accuracy using
1,600 filters, which is competitive to the best publishedsingle layer result (59.0%).
5.4. Phone classification

Table 4. Phone classification accuracy on the TIMIT core
test set using linear SVMs.
Methods (Linear SVM)
MFCC
sparse RBM
sparse TIRBM
sparse coding (Ngiam et al., 2011)sparse filtering (Ngiam et al., 2011)

Table 5. Phone classification accuracy on the TIMIT core
test set using RBF-kernel SVMs.
Methods (RBF-kernel SVM)
MFCC (baseline)
MFCC + TIRBM (K= 256)
MFCC + TIRBM (K= 512)
MFCC + SC (Ngiam et al., 2011)
MFCC + SF (Ngiam et al., 2011)
MFCC + CDBN (Lee et al., 2009)
H-LMGMM (Chang & Glass, 2007)

Accuracy
80.0%
81.0%
81.5%
80.1%
80.5%
80.3%
81.3%

accuracy on the TIMIT dataset. By following (Ngiamet al., 2011), we generated 39-dimensional MFCC features and used 11 contiguous frames of them as an
input patch. For TIRBMs, we applied three temporaltranslations with the stride of 1 frame.
First, we compared the classification accuracy using
the linear SVM to evaluate the performance gain coming from the unsupervised learning algorithms, by following the experimental setup in (Ngiam et al., 2011).8
As reported in Table 4, the TIRBM showed an improvement over the sparse RBM, as well as the sparse
coding and sparse filtering.
In the next setting, we used the RBF-kernel SVM(Chang & Lin, 2011) on the extracted features thatare concatenated with MFCC features. We used default RBF kernel width for all experiments and performed cross-validation on the C values. As shown in
Table 5, combining MFCC with TIRBM features was
the most effective and resulted in 1% improvement inclassification accuracy over the baseline MFCC features. By increasing the number of TIRBM features
to 512, we were able to beat the best published resultson the TIMIT phone classification tasks using hierarchical LM-GMM classifier (Chang & Glass, 2007).

6. Conclusion and Future WorkIn this work, we proposed novel feature learning algorithms that can achieve invariance to the set of predefined transformations. Our method can handle general transformations (e.g., translation, rotation, and8

To show the broad applicability of our method to otherdata types, we report the 39-way phone classification

Accuracy
68.2%
76.3%
77.6%
76.8%
75.7%

We used K = 256 with a two-sided encoding function by using the positive and negative weight matrices[W, −W], as suggested by Ngiam et al. (2011).

Learning Invariant Representations with Local Transformations

scaling), and we experimentally showed that learning invariant features for such transformations leads
to strong classification performance. In future work,
we plan to work on learning transformations from thedata and combine it with our algorithm. By automatically learning transformation matrices from the data,
we will be able to learn more robust features, which
will potentially lead to significantly better feature representations.
Acknowledgments
This work was supported in part by a Google Faculty
Research Award.

References
Bengio, Y., Lamblin, P., Popovici, D., and Larochelle, H.
Greedy layer-wise training of deep networks. In NIPS,
2007.
Chang, Chih-Chung and Lin, Chih-Jen. LIBSVM: A library for support vector machines. ACM Transactions
on Intelligent Systems and Technology, 2:27:1–27:27,
2011. http://www.csie.ntu.edu.tw/~cjlin/libsvm.
Chang, H. and Glass, J. R. Hierarchical large-margin gaussian mixture models for phonetic classication. In ASRU,
2007.
Ciresan, D. C., Meier, U., Masci, J., Gambardella, L. M.,
and Schmidhuber, J. High-performance neural networks
for visual object classification. Technical report, 2011.
Coates, A. and Ng, A. Y. The importance of encoding versus training with sparse coding and vector quantization.
In ICML, 2011a.

Kavukcuoglu, K., Sermanet, P., Boureau, Y. L., Gregor,K., Mathieu, M., and LeCun, Y. Learning convolutionalfeature hierachies for visual recognition. In NIPS, 2010.
Kivinen, J. and Williams, C. Transformation equivariant
boltzmann machines. In ICANN, 2011.
Krizhevsky, A. Learning multiple layers of features fromTiny Images. Master’s thesis, University of Toronto,
2009.
Krizhevsky, A. Convolutional deep belief networks on cifar10. Technical report, 2010.
Larochelle, H., Erhan, D., Courville, A., Bergstra, J., andBengio, Y. An empirical evaluation of deep architectureson problems with many factors of variation. In ICML,
2007.
LeCun, Y., Boser, B., Denker, J. S., Henderson, D.,
Howard, R. E., Hubbard, W., and Jackel, L. D. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1:541–551, 1989.
Lee, H., Ekanadham, C., and Ng, A. Y. Sparse deep belief
network model for visual area V2. In NIPS, 2008.
Lee, H., Largman, Y., Pham, P., and Ng, A. Y. Unsupervised feature learning for audio classification using
convolutional deep belief networks. In NIPS. 2009.
Lee, H., Grosse, R., Ranganath, R., and Ng, A. Y. Unsupervised learning of hierarchical representations withconvolutional deep belief networks. Communications of
the ACM, 54(10):95–103, 2011.
Lowe, D. Object recognition from local scale-invariant features. In ICCV, 1999.
Ngiam, J., Koh, P. W., Chen, Z., Bhaskar, S., and Ng,
A. Y. Sparse filtering. In NIPS, 2011.

Coates, A. and Ng, A. Y. Selecting receptive fields in deep
networks. In NIPS, 2011b.

Ranzato, M., Poultney, C., Chopra, S., and LeCun, Y. Efficient learning of sparse representations with an energybased model. In NIPS, 2006.

Coates, A., Lee, H., and Ng, A. Y. An analysis of singlelayer networks in unsupervised feature learning. In AISTATS, 2011.

Sohn, K., Jung, D. Y., Lee, H., and Hero III, A. Efficientlearning of sparse, distributed, convolutional feature representations for object recognition. In ICCV, 2011.

Hinton, G. E. Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8):1771–1800, 2002.

van Hateren, J. H. and van der Schaaf, A. Independent
component filters of natural images compared with simple cells in primary visual cortex. Proceedings of theRoyal Society B, 265:359–366, 1998.

Hinton, G. E. and Salakhutdinov, R. Reducing the dimensionality of data with neural networks. Science, 313(5786):504–507, 2006.
Hinton, G. E., Osindero, S., and Teh, Y.-W. A fast learningalgorithm for deep belief nets. Neural Computation, 18(7):1527–1554, 2006.
Hyvärinen, A., Hoyer, P. O., and Inki, M. O. Topographic
independent component analysis. Neural Computation,
13(7):1527–1558, 2001.
Kavukcuoglu, K., Ranzato, M., Fergus, R., and LeCun, Y.
Learning invariant features through topographic filter
maps. In CVPR, 2009.

Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.A.
Extracting and composing robust features with denoising autoencoders. In ICML, 2008.
Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., andManzagol, P. A. Stacked denoising autoencoders: Learning useful representations in a deep network with a local
denoising criterion. JMLR, 11:3371–3408, 2010.
Yu, K., Lin, Y., and Lafferty, J. Learning image representations from the pixel level via hierarchical sparse coding.
In CVPR, 2011.
Zeiler, M. D., Krishnan, D., Taylor, G. W., and Fergus, R.
Deconvolutional networks. In CVPR, 2010.