Why does the unsupervised pretraining encourage moderate-sparseness?
Authors: Jun Li, Wei Luo, Jian Yang, Xiaotong Yuan
ICLR 2014

Abstract: It is well known that direct training of deep multi-layer neural networks (DNNs) will generally lead to poor results. A major progress in recent years is the invention of various unsupervised pretraining methods to initialize network parameters and it was shown that such methods lead to good prediction performance. However, the reason for the success of the pretraining has not been fully understood, although it was argued that regularization and better optimization play certain roles. This paper provides another explanation for the effectiveness of the pretraining, where we empirically show the pretraining leads to a higher level of sparseness of hidden unit activation in the resulting neural networks, and the higher sparseness is positively correlated to faster training speed and better prediction accuracy. Moreover, we also show that rectified linear units (ReLU) can capture the sparseness benefits of the pretraining. Our implementation of DNNs with ReLU does not require the pretraining, but achieves comparable or better prediction performance than traditional DNNs with pretraining on standard benchmark datasets.


Peer review:
[Review]:
Brief summary of paper: 
The paper is an experimental evaluation of the sparsity levels of representations in deep networks learned through several approaches. These include 3 types of non-linearity (sigmoid, tanh and ReLU), with and without pretraining, and dropout. The main findings/conclusions are that pretraining yields a moderate level of representation sparsity, and that using their brand of purely supervised ReLU with dropout yields even sparser solutions and better classification on MNIST/CIFAR10/NORB.

Assessment:
Overall I see very little originality in this paper, the conclusions are not really novel, and the paper is very unclearly written. Contrary to its title it does not bring any insight as to '*WHY*  unsupervised pretraining encourages moderate sparseness'. A thorough and clearly presented empirical study of the sparsity properties of representations learned by different algorithms could have been interesting. But unfortunately, in its current state this work is extremely unclearly written, with confusing terminology, vague reasoning, poor structure, and insufficiently described experimental setup. The classification performance obtained with 'their ReLU' appears rather good, but the paper does not investigate in depth what would explain the improvement over the other ReLU results from the literature they report. If their approach/implementation indeed has an advantage, its specificity should be explained and evaluated in detail. It is unclear from the writeup whether hyper-parameters were chosen/tuned using proper methodology.

Pros & Cons:
- lack of originality
- very unclear, confusing, writeup