k-Sparse Autoencoders
Authors: Alireza Makhzani, Brendan Frey
ICLR 2014

Abstract: Recently, it has been observed that when representations are learnt in a way that encourages sparsity, improved performance is obtained on classification tasks. These methods involve combinations of activation functions, sampling steps and different kinds of penalties. To investigate the effectiveness of sparsity by itself, we propose the k-sparse autoencoder, which is a linear model, but where in hidden layers only the k highest activities are kept. When applied to the MNIST and NORB datasets, we find that this method achieves better classification results than denoising autoencoders, networks trained with dropout, and restricted Boltzmann machines. k-sparse autoencoders are simple to train and the encoding stage is very fast, making them well-suited to large problem sizes, where conventional sparse coding algorithms cannot be applied.


Peer review:
[Review]:
The authors propose an auto encoder with linear encoder and decoder, but with sparsity that keeps only k elements in the hidden layer nonzero. They show that it works as well or better then more complicated methods.

Novelty: Simple but works
Quality: Good

Details:
- The paper introduces a very simple idea which I am sure many people not only thought of but implemented, including me. However the main point here is that the authors actually made it work well and made a connection to a sparse coding algorithm. One of tricks of making it work seems to be to start with a large number of allowed nonzero elements and then decrease it, otherwise, many filters would not ever be used. 

- Is there a mistake in the algorithm box as presented x = Wz+b'. Shouldn't the z be replaced by something like z_Gamma where the latter is obtained from z by setting elements that are not in the group of k largest to zero? Because that's what the description in the rest of the paper implies, for example in 2.2.

- Table - it would be good to explain that the net is in Table 3's caption.