Auto-pooling: Learning to Improve Invariance of Image Features from Image Sequences
Authors: Sainbayar Sukhbaatar, Takaki Makino, Kazuyuki Aihara
ICLR 2013

Abstract: Learning invariant representations from images is one of the hardest challenges facing computer vision. Spatial pooling is widely used to create invariance to spatial shifting, but it is restricted to convolutional models. In this paper, we propose a novel pooling method that can learn soft clustering of features from image sequences. It is trained to improve the temporal coherence of features, while keeping the information loss at minimum. Our method does not use spatial information, so it can be used with non-convolutional models too. Experiments on images extracted from natural videos showed that our method can cluster similar features together. When trained by convolutional features, auto-pooling outperformed traditional spatial pooling on an image classification task, even though it does not use the spatial topology of features.

arXiv:1301.3323v4  [cs.CV]  18 Mar 2013
Auto-pooling: Learning to Improve Invariance of
Image Features from Image Sequences
Sainbayar Sukhbaatar
Dept. of Mathematical Informatics, IST
The University of Tokyo
Tokyo 116-8656, Japan
sainaa@sat.t.u-tokyo.ac.jp
Takaki Makino Kazuyuki Aihara
Institute of Industrial Science
The University of Tokyo
Tokyo 153-8505, Japan
{mak,aihara}@sat.t.u-tokyo.ac.jp
Abstract
Learning invariant representations from images is one of th e hardest challenges
facing computer vision. Spatial pooling is widely used to create invariance to spa-
tial shifting, but it is restricted to convolutional models. In this paper, we propose
a novel pooling method that can learn soft clustering of feat ures from image se-
quences. It is trained to improve the temporal coherence of features, while keeping
the information loss at minimum. Our method does not use spat ial information,
so it can be used with non-convolutional models too. Experim ents on images ex-
tracted from natural videos showed that our method can clust er similar features
together. When trained by convolutional features, auto-pooling outperformed tra-
ditional spatial pooling on an image classiﬁcation task, ev en though it does not
use the spatial topology of features.
1 Introduction
The main difﬁculty of object recognition is that the appearance of an object can change in complex
ways. To build a robust computer vision, one needs represent ations that are invariant to various
transformations. The concept of invariant features dates b ack to Hubel and Wiesel’s seminal work
[7], in which cells in a cat’s visual cortex are studied. Theyfound two types of cells: simple cells that
responded to a speciﬁc pattern at a speciﬁc location, and complex cells that showed more invariance
to location and orientation.
Inspired by simple and complex cells, the spatial pooling st ep is introduced to computer vision
architectures along with the convolution step [4, 12, 13]. In the convolution step, the same feature is
applied to different locations. Then in the pooling step, responses from nearby locations are pooled
together (typically with a sum or a max operation) to create i nvariance to small spatial shifting.
However, spatial pooling only works with convolutional models. Also, spatial pooling only improves
the invariance to spatial shifting.
An ideal pooling should make features invariant to major typ es of transformations that appear in
the nature. For example, to distinguish people from other ob jects, one need a representation that
is invariant to various transformations of the human body. T he complexity of such transformations
creates the necessity of more adaptive pooling that does not rely on manual ﬁxed conﬁgurations.
One promising way to obtain such an adaptive pooling is an unsupervised learning approach.
In recent years, more adaptive spatial pooling methods have been proposed. Jia et al. [9] showed
that it is possible to learn custom pooling regions speciali zed for a given classiﬁcation task. The
training starts with many pooling region candidates, but on ly a few of them are used in the ﬁnal
classiﬁcation. This selection of pooling regions is achieved by supervised learning incorporated into
the training of a classiﬁer. Although this method learns pooling regions from data, it is still restricted
1
spatial pooling auto-pooling
low-level
features
pooled
representation
Figure 1: In spatial pooling, low-level features are “hard” clustered by their spatial organization.
Auto-pooling, on the other hand, learns “soft” clustering of low-level features from image sequences.
to spatial shifting. Further, it is not suited for deep learn ing, where lower layers are trained in an
unsupervised way.
Another method that improves spatial pooling is proposed by Coates and Ng [3], in which local
features are clustered by their similarity. A similarity me tric between features is deﬁned from their
energy correlations. Then, nearby features from the same cl uster are pooled together to create rota-
tion invariance. However, the invariance to spatial shifting is still achieved through the same spatial
pooling, which restricts this model to convolutional models.
Beside from spatial pooling, there are methods [8, 10, 16] th at create invariance by placing fea-
tures on a two-dimensional topographic map. During training, nearby features are constrained to be
similar to each other. Then, invariant representations are achieved by clustering features in a local
neighborhood. However, those methods ﬁx clustering manually, which restricts clusters from having
adaptive sizes that depend on the nature of their features. Also, we cannot guarantee that the optimal
feature clustering can be mapped into two-dimensional spac e. For example, edge detectors have at
least four dimensions of variation, so an ideal clustering c an be achieved by placing edge detectors
in a four-dimensional space and grouping nearby features. I t will be difﬁcult to approximate such
a clustering with a two-dimensional map. Moreover, those me thods cannot be used with features
already learned by another model.
Slowness has been used in many methods as a criterion for inva riant features [1, 15, 2, 6]. The
intuition is that if a feature is invariant to various transformations, then its activation should change
slowly when presented with an image sequence containing those transformations. Mobahi et al. [15]
incorporated unsupervised slowness learning with supervised back-propagation learning, which im-
proved the classiﬁcation rate. However, our focus is a simpl e unsupervised method that can make
features invariant without changing them, so it can easily replace and improve spatial pooling in any
application.
In this paper, we proposeauto-pooling, a novel pooling method that learns soft clustering of features
from image sequences in an unsupervised way. Our method impr oves the invariance of features
using temporal coherence of image sequences. Two consecuti ve frames are likely to contain the
same object, so auto-pooling minimizes the distance betwee n their pooled representations. At the
same time, the information loss due to pooling is also minimi zed. This is done by minimizing the
reconstruction error in the same way as auto-encoders [17, 18]. Through experiments, we show that
our method can pool similar features and increase accuracy of an image classiﬁcation task.
There are several advantages in auto-pooling over traditio nal spatial pooling (see Figure 1). First,
it produces invariance to all types of transformations that present in natural videos. Second, auto-
pooling is a more biologically plausible model for complex cells because its parameters are learned
from image sequences rather than being manually deﬁned. Thi rd, auto-pooling can be used with
non-convolutional models because it does not use spatial information.
2 Auto-pooling
Auto-pooling is a pooling method that learns transformatio ns appeared in image sequences. It is
trained by image sequences in an unsupervised way to make fea tures more invariant. The goal of
training is to cluster similar features together, so that small transformations would not affect pooled
representations. Two features are considered similar if they are traversable by a small transformation
such as shift or rotation. We use natural videos as the resour ce for learning similarity between
2
yy
x
z
P
y’y’
x’
z’
P
invariance cost
reconstruction cost
input image
P
T
P
T
pooled
representation
auto-pooling
feature
extraction
Figure 2: Structure of an auto-pooling model
features, because they are rich in various complex transformations. Moreover, image sequences are
available to animals and humans as early as their birth, so it is biologically plausible to use them in
learning of complex-cell-like invariant features.
We believe that there are two desirable properties in good po oling methods. The ﬁrst property is
that if two images show the same object, then their pooled rep resentations should be the same.
Auto-pooling tries to meet this invariance property by minimizing the distance between pooled rep-
resentations of consecutive frames, which are likely to contain the same object. The second desirable
property is that the information loss due to pooling should b e minimal. This is the same as max-
imizing the cross entropy between inputs and their pooled re presentations. This entropy property
could be obtained by minimizing the error between inputs and reconstructions from their pooled
representations.
Instead of image sequences, it is convenient to use image pairs taken from consecutive video frames
as training data. Such image pairs can be written as
X = {x1, x′
1, x2, x′
2, ..., xN , x′
N }.
If an object is present in image xi, then it is likely to be present in x′
i too, because frames xi and x′
i
have very small time difference (only 33ms for videos with 30 frames per second). Let us assume
that the low-level feature extraction is done by
yi = f(xi), y′
i = f(x′
i).
Here, f can be any function as long as yi, y′
i are non-negative.
In auto-pooling, clustering of features is parameterized b y a pooling matrix P . We require all
elements of P to be non-negative because they represent the associations between features and
clusters. Pij = 0 means that j-th feature does not belong to i-th cluster. On the other hand,
large Pij indicates that i-th cluster contains j-th feature. Then, pooling is done by a simple matrix
multiplication, that is
zi = P yi, z′
i = P y′
i.
If the dimension of feature vectors yi, y′
i is M, and the dimension of pooled representations zi, z′
i
is K, then P is a K × N matrix. While in spatial pooling, elements of P are ﬁxed to 0 or 1 based on
the topology of feature maps, auto-pooling generalizes it b y allowing Pij to take any non-negative
value.
Our main goal is to learn pooling parameters Pij from data, without using the explicit spatial infor-
mation. Training of auto-pooling is driven by two cost functions. The ﬁrst cost function
J1 = 1
N
N∑
i=1
1
2∥zi − z′
i∥2
2
is for the invariance property, and minimizes the distance between pooled representations zi and z′
i.
However, there is a trivial solution ofP = 0if we use only this cost function.
The second cost function corresponds to the entropy propert y, and encourages pooled representa-
tions to be more informative of their inputs. Input yi and y′
i are reconstructed from their pooled
3
representations by
ˆyi = P T zi, ˆy′
i = P T z′
i
using the same parameters as the pooling step. Then, the reco nstruction error is minimized by the
cost function of
J2 = 1
N
N∑
i=1
1
2(∥yi − ˆyi∥2
2 + ∥y′
i − ˆy′
i∥2
2).
This prevents auto-pooling from throwing away too much information for the sake of invariance.
Auto-pooling is similar to auto-encoders, which are used in feature learning. In auto-pooling, the
input is reconstructed in the same way as auto-encoders. Als o, the reconstruction cost function
J2 is exactly same as the cost function of auto-encoders. Howev er, there are several important
differences between them. First, parameters of auto-pooli ng are restricted to non-negative values.
Second, activation functions of auto-pooling are linear and have no biases. Third, auto-pooling has
an additional cost function for temporal coherence.
The ﬁnal cost function of auto-pooling is
J = λJ1 + J2,
where the parameter λ ≥ 0 controls the weight of the invariance cost function. Larger λ will make
features more invariant by discarding more information. The training is done by minimizing the cost
function with a simple gradient descent algorithm.
2.1 Invariance Score
For evaluating our pooling model, we deﬁne a score for measur ing invariance of features (a sim-
ilar score is also introduced in [5]). A simple measurement o f feature’s invariance is its average
activation change between two neighboring frames, which is
G = 1
N
N∑
n=1
∥g(xn) − g(x′
n)∥2.
Here, g(x) := f(x) if we are measuring invariance of raw features, and g(x) := P f (x) if we are
measuring invariance of pooled representations.
For invariant features, G should be small. However, features can cheat by simply making its activa-
tion constant to reduce G , which is obviously not useful. An ideal invariant feature should take the
same value only if the stimuli are from consecutive frames. F or frames chosen from random tim-
ings, an invariant feature should have different activitie s because it is likely that the inputs contain
different objects. Therefore, the average distance between two random frames
H = 1
N
N∑
n=1
∥g(xn) − g(x′
σ (n))∥2
should be large for invariant features. Here σ is a random permutation of {1, 2, ..., N }. The invari-
ance score is deﬁned as
F = H
G , (1)
which will be large only if a feature is truly invariant.
3 Experiments
We will show the effectiveness of our method with two types ofexperiments. In the ﬁrst experiment,
we train an auto-pooling model with non-convolutional feat ures. The goal of this experiment is to
see whether similar features are being pooled together. We a lso measured the invariance score
of features before and after pooling. In the second experime nt, we compared our method with
traditional spatial pooling on an image classiﬁcation task.
4
Figure 3: Patch pairs (each column) extracted from natural videos
3.1 Clustering of Image Features
The goal of this set of experiments is to analyze feature clus ters learned by auto-pooling. We pre-
pared a dataset of16×16 gray-scale patch pairs from natural videos1. Patch pairs are extracted from
random locations of consecutive frames. Some of the patch pairs are shown in Figure 3.
Figure 4: Clusters (shown in each column) of features learned from the patch pair dataset
We used a sparse auto-encoder to learn 400 features from patches. Then, we trained an auto-pooling
model on those features. Since auto-pooling performs soft c lustering, it is hard to visualize the
clustering result. For simplicity, we used a small threshol d to show some of the learned clusters in
Figure 4, where each column represents a single cluster. For i-th cluster, we showed features with
Pij > ε. It is evident that similar features are clustered together. Also, one can see that the size of a
cluster vary depending on the nature of its features.
Figure 5: Diversity of edge detectors in a single cluster
To display clusters more clearly, some clusters of edge detectors are shown in more detail in Figure 5,
in which edge detectors are replaced by corresponding thin lines. This allows us to see the diversity
of edge detectors inside each cluster. The important thing i s that there is variance in orientations
1We used 44 short videos. All videos are obtained from http://www.vimeo.com and had the Creative
Commons license. Although we tried to include videos with th e same objects as CIFAR10 (a labeled image
dataset used in the next experiment), image patches extracted from the videos were qualitatively different than
images from CIFAR10. Even if a video contains a desired objec t, not all frames show the object. Also, most
patches only included a part of an object, because the patch size was much smaller than the frame size,
5
as well as in locations, which means that auto-pooling can cr eate representations invariant to small
rotations.
    




λ
,QYDULDQFHb6FRUH 
b
b
SRROHGbIHDWXUHV
UDZbIHDWXUHV
Figure 6: Invariance scores before and after pooling
Next, we analyzed the effect of pooling on image features using our invariance score. Figure 6 shows
invariance scores measured at various values of λ, which controls the weight of the invariance cost.
The invariance score is signiﬁcantly improved by the poolin g, especially for large λ values. It is
not surprising because larger λ puts more importance on the invariance cost, thus makes pool ed
representations less likely to change between consecutive frames. As a result, G in equation 1
becomes smaller and increases the invariance cost.
At the same time, however, increase of λ diminishes the role of the reconstruction cost, which
was preventing the loss of too much information. Too large λ makes pooled representations over-
invariant, having constant values all the time. This results in a small H and decreases the invariance
cost. This side effect of large λ can be observed from Figure 6, where the invariance score stopped
its increase at large λ.
3.2 Image Classiﬁcation
Figure 7: Samples from the patch pair dataset used in this experiment
Next, we tested the effectiveness of our pooling method by applying it to an image classiﬁcation task.
We used two datasets. For image classiﬁcation, we used CIFAR 10 dataset [11], which contains 50
thousand labeled images from ten categories. All images wer e 32×32 pixels in size, and had three-
color channels. For training of an auto-pooling model, we pr epared a patch pair dataset in the
same way as the previous experiment, except patches were 32×32 color images to match CIFAR10
images. Some samples from the patch pair dataset are shown in Figure 7.
Figure 8: Features learned by a sparse auto-encoder from small image patches
In the feature extraction step, we used a convolutional model. We trained a sparse auto-encoder with
100 hidden units on 6×6 small patches extracted from CIFAR10 images. Learned local features are
shown in Figure 8. Then, those 6 × 6 local features are duplicated to all possible locations of 32×32
images, resulting in 100 feature maps of 27 × 27.
6
Figure 9: Feature clusters learned by an auto-pooling model . The right side shows pooled regions
on feature maps, and the left side shows corresponding local features.
The convolutional feature extraction produced very large ( exceeding 100 gigabytes) training data
for auto-pooling. Luckily, the training took only few hours because we implemented our algorithm
on a graphic card (Tesla K20c) using CUDAMat library [14]. So me of the learned feature clusters
are visualized in Figure 9. For each cluster, we showed 15 fea ture maps with the largest pooling
area (i.e. maxk(∑
j∈Sk
Pij), where Sk is the set of features in k-th feature map). Pij = 0is shown
in gray and large Pij is shown in white. Local features corresponding to the featu re maps are also
shown in Figure 9.
Unlike spatial pooling, each cluster learned by auto-pooli ng extended to multiple feature maps.
Pooling regions (i.e., white areas in Figure 9) of those maps usually have the same continues spatial
distribution, which will create spatial invariance in the s ame way as spatial pooling. If we observe
those pooling regions carefully, however, we can see the sma ll variance in their locations. This
location variance is inversely related to the location vari ance of corresponding local features. For
example, if there is a edge detector in the lower part of a 6 × 6 local feature, corresponding pooling
regions will have upper position in 27 × 27 feature maps.
Beside from pooling by spatial areas, auto-pooling also suc ceeded in clustering similar local fea-
tures. In some clusters, edge detectors of similar orientat ions are grouped together. This will make
pooled representations invariant to small rotations, whic h is a clear advantage over traditional spa-
tial pooling. In addition, clustering of local features onl y differing in their locations will reduce the
redundancy created by convolutional feature extraction.
We compared our pooling method with traditional spatial pooling on a classiﬁcation task, in which
a supervised classiﬁer is trained by pooled representation s of labeled images. For auto-pooling, we
varied the number of clusters from 400 to 2500. For spatial po oling, we can only change the grid
size. However, it is possible to use multiple spatial poolin g at once [12] to produce better results.
We denote a spatial pooling that used 2 × 2 and 3 × 3 grids by 2 × 2 + 3× 3.
Table 1: Classiﬁcation accuracy on CIFAR10 (AP=auto-pooling, SP=spatial pooling)
Accuracy Accuracy
Pooling methods (full) (small)
AP (400 clusters) 64.6% 61.6%
AP (800 clusters) 67.2% 62.9%
AP (1300 clusters) 69.0% 63.8%
AP (1600 clusters) 69.4% 64.3%
AP (2000 clusters) 69.7% 65.0%
AP (2500 clusters) 69.3% 63.5%
SP (2 × 2) 63.8% 60.7%
SP (3 × 3) 68.2% 61.5%
SP (2 × 2 + 3× 3) 68.2% 61.2%
SP (4 × 4) 68.4% 58.7%
SP (2 × 2 + 3× 3 + 4× 4) 68.2% 57.6%
7
500 1000 1500 2000 2500 300056
58
60
62
64
66
68
70
the number of clusters
accuracy
auto−pooling (full)
auto−pooling (small)
spatial pooling (full)
spatial pooling (small)
Figure 10: Classiﬁcation accuracies are plotted as functions of the number of features
In classiﬁcation, we trained a linear SVM with pooled repres entations. The results are shown in
Table 1. We trained the classiﬁer with two training data: a full data with 5000 examples per class, and
a smaller one with 1000 examples per class. Since the number o f features is an important factor in
classiﬁcation, we plotted the accuracy of the two pooling methods against the number of clusters in
Figure 10. Auto-pooling outperformed traditional spatialpooling for most of the time. Especially for
small training data, the difference between the two pooling methods was substantial. This indicates
that auto-pooling is better at generalization, which is themain goal of invariant features. The spatial
pooling, on other hand, shows the sign of over-ﬁtting when its pooling regions are increased.
4 Conclusions
In this paper, we introduced auto-pooling, a novel pooling m ethod that can generalize traditional
spatial pooling to transformations other than spatial shif ting. Auto-pooling tries to make features
more temporally coherent, having slow changing activation when presented with a continues image
sequence. The information loss due to pooling is kept minimum using the same cost function as auto-
encoders. The main advantage of our method is that it learns t o cluster features from data, rather
than relying on manual heuristic spatial divisions. Theref ore, auto-pooling is a more biologically
plausible model for complex cells.
When trained by image pairs extracted from natural videos, a uto-pooling successfully clustered
similar features together. We showed that such clustering could signiﬁcantly improve the invariance
of features. Also, our pooling model was more effective than traditional spatial pooling when it
was used in a real-world classiﬁcation task, where spatial pooling had the advantage of using spatial
information of features.
In our experiments, the advantage of auto-pooling over spatial pooling was mainly restricted to learn-
ing of rotation invariance. This is because auto-pooling is applied to low-level features, which were
mostly edge detectors with the size. Therefore, the only pos sible variance beside spatial shifting
was rotation. We believe that if we use auto-pooling insteadof spatial pooling in deep architectures,
we can create invariance to more complex transformations su ch as three-dimensional rotations and
distortions.
Acknowledgments
This research is supported by the Aihara Innovative Mathema tical Modelling Project, the Japan
Society for the Promotion of Science (JSPS) through the “Fun ding Program for World-Leading
8
Innovative R&D on Science and Technology (FIRST Program),”initiated by the Council for Science
and Technology Policy (CSTP).
References
[1] P. Berkes and L. Wiskott. Slow feature analysis yields a rich repertoire of complex cell properties.Journal
of Vision, 5(6), 2005.
[2] C. Cadieu and B. Olshausen. Learning transformational i nvariants from natural movies. Advances in
Neural Information Processing Systems 21, pages 209–216, 2009.
[3] A. Coates and A. Ng. Selecting receptive ﬁelds in deep net works. In Advances in Neural Information
Processing Systems 24, pages 2528–2536. 2011.
[4] K. Fukushima. Neocognitron: A self-organizing neural n etwork model for a mechanism of pattern recog-
nition unaffected by shift in position. Biological Cybernetics, 36(4):193–202, 1980.
[5] I. Goodfellow, Q. Le, A. Saxe, H. Lee, and A. Ng. Measuring invariances in deep networks. In Advances
in Neural Information Processing Systems 22, pages 646–654. 2009.
[6] K. Gregor and Y . LeCun. Emergence of complex-like cells i n a temporal product network with local
receptive ﬁelds. arXiv preprint arXiv:1006.0448, 2010.
[7] D. Hubel and T. Wiesel. Receptive ﬁelds, binocular inter action and functional architecture in the cat’s
visual cortex. The Journal of Physiology, 160(1):106, 1962.
[8] A. Hyv¨ arinen, P. Hoyer, and M. Inki. Topographic independent component analysis. Neural Computation,
13(7):1527–1558, 2001.
[9] Y . Jia, C. Huang, and T. Darrell. Beyond spatial pyramids : Receptive ﬁeld learning for pooled image
features. In Computer Vision and Pattern Recognition, 2012 IEEE Confere nce on, pages 3370–3377.
IEEE, 2012.
[10] K. Kavukcuoglu, M. Ranzato, R. Fergus, and Y . LeCun. Learning invariant features through topographic
ﬁlter maps. In Computer Vision and Pattern Recognition, 2009 IEEE Conference on, pages 1605–1612.
IEEE, 2009.
[11] A. Krizhevsky and G. Hinton. Learning multiple layers o f features from tiny images. Master’s thesis,
Department of Computer Science, University of Toronto, 2009.
[12] S. Lazebnik, C. Schmid, and J. Ponce. Beyond bags of features: Spatial pyramid matching for recognizing
natural scene categories. In Computer Vision and Pattern Recognition, 2006 IEEE Conference on, pages
2169–2178. IEEE, 2006.
[13] Y . LeCun, L. Bottou, Y . Bengio, and P. Haffner. Gradient-based learning applied to document recognition.
Proceedings of the IEEE, 86(11):2278–2324, 1998.
[14] V . Mnih. CUDAMat: a CUDA-based matrix class for Python. Technical report, Department of Computer
Science, University of Toronto, 2009.
[15] H. Mobahi, R. Collobert, and J. Weston. Deep learning from temporal coherence in video. InProceedings
of the 26th International Conference on Machine Learning, pages 737–744. ACM, 2009.
[16] S. Osindero, M. Welling, and G. Hinton. Topographic pro duct models applied to natural scene statistics.
Neural Computation, 18(2):381–414, 2006.
[17] M. Ranzato, S. Chopra, and Y . LeCun. Efﬁcient learning o f sparse representations with an energy-based
model. In Advances in Neural Information Processing Systems 19, pages 1137–1144. MIT Press, 2007.
[18] P. Vincent, H. Larochelle, Y . Bengio, and P.A. Manzagol . Extracting and composing robust features
with denoising autoencoders. In Proceedings of the 25th International Conference on Machine Learning,
pages 1096–1103. ACM, 2008.
9


Peer review:
[Review]:
First off, let me say thank you for citing my + my co-authors' paper on measuring invariances.

I have a few thoughts about invariance and temporal coherence that I hope you might find helpful.

Regarding invariance, I think that invariance is not such a great property on its own. What you really want is to disentangle the different factors of variation in the dataset. Invariance plays a role in this process, because if you want one feature to correspond only to one factor of variation, it must be invariant to all of the others. But it's really a very small part of the picture. For the purposes of our paper on measuring invariances, invariance was a good enough proxy for disentangling that we could use it test the hypothesis that deep learning systems become more invariant with depth. But I don't think invariance is a good enough property to serve as the main part of your objective function.

Regarding temporal coherence, I think a common mistake people make is to jump from the idea that features should be 'coherent' to the idea that features should be 'slow.' I think that useful features are spread over a wide spectrum of timescales. It's true that the fastest varying features are probably just noise. But the slowest varying features are probably not especially useful either. For example, if you put a camera on a streetcorner, the amount of sunlight in the scene would usually change slower than the identities of the people in the scene. I think probably the way to make progress with applications of temporal coherence is to study new ways of encouraging features to be coherent rather than just slow.

Some general suggestions on how to improve your results: You should read Adam Coates' ICML 2011 paper, which is about finding the best training algorithm and feature encoding method for single-layer architectures. I think if you use larger dictionaries (1600 instead of 100), train using OMP-1 or sparse coding instead of sparse autoencoders, and extract using T-encoding you will do much better and have a shot at beating state of the art, or at least beating Jia and Huang. Adam Coates' work shows that sparse autoencoders don't make very good feature extractors, and also that small dictionaries don't perform very well, so you're really hurting your numbers by using that setup as your feature extractor.

Finally, I think you're missing a few references. In particular, your approach is very closely related to Slow Feature Analysis, so you should cite Laurenz Wiskott and comment on the similarities.