k-Sparse Autoencoders
Authors: Alireza Makhzani, Brendan Frey
ICLR 2014

Abstract: Recently, it has been observed that when representations are learnt in a way that encourages sparsity, improved performance is obtained on classification tasks. These methods involve combinations of activation functions, sampling steps and different kinds of penalties. To investigate the effectiveness of sparsity by itself, we propose the k-sparse autoencoder, which is a linear model, but where in hidden layers only the k highest activities are kept. When applied to the MNIST and NORB datasets, we find that this method achieves better classification results than denoising autoencoders, networks trained with dropout, and restricted Boltzmann machines. k-sparse autoencoders are simple to train and the encoding stage is very fast, making them well-suited to large problem sizes, where conventional sparse coding algorithms cannot be applied.


Peer review:
[Review]:
I have not read the paper that carefully.  But the idea of the k-sparse autoencoder seems very similar to the Orthogonal Matching Pursuit (OMP-k) training and encoding used in Coates and Ng (http://www.stanford.edu/~acoates/papers/coatesng_icml_2011.pdf).

The difference, it seems to me, is that OMP-k allows less than k units to be active, and also the scheduling idea (4.2.1), and the alpha multiplier for the encoding stage.

Is there something else I am missing that distinguishes your approach?  

Otherwise, I would like to see comparisons to OMP-k, and to a simple threshold encoding approach as in Coates and Ng.