Why does the unsupervised pretraining encourage moderate-sparseness?
Authors: Jun Li, Wei Luo, Jian Yang, Xiaotong Yuan
ICLR 2014

Abstract: It is well known that direct training of deep multi-layer neural networks (DNNs) will generally lead to poor results. A major progress in recent years is the invention of various unsupervised pretraining methods to initialize network parameters and it was shown that such methods lead to good prediction performance. However, the reason for the success of the pretraining has not been fully understood, although it was argued that regularization and better optimization play certain roles. This paper provides another explanation for the effectiveness of the pretraining, where we empirically show the pretraining leads to a higher level of sparseness of hidden unit activation in the resulting neural networks, and the higher sparseness is positively correlated to faster training speed and better prediction accuracy. Moreover, we also show that rectified linear units (ReLU) can capture the sparseness benefits of the pretraining. Our implementation of DNNs with ReLU does not require the pretraining, but achieves comparable or better prediction performance than traditional DNNs with pretraining on standard benchmark datasets.


Peer review:
[Review]:
Review Summary
This work empirically show the unsupervised pretraining encourage moderate-sparseness in a deep neural network, which could lead better classification performance. The author also proposed that ReLU DNNs without pretraining could also lead to moderate-sparseness.  
Though empirically studying sparseness in DNNs is interesting and important, this submission could potentially be improved. 

Pros
-- well-written and organized
-- an interesting idea for exploring the role of unsupervised pretraining in deep learning
Cons
-- I am not convinced that the unsupervised pretraining itself indeed encourage sparseness in higher layers of networks. As mentioned in your paper, those pretraining algorithms try to decrease the reconstruction errors under some constrains. If those models were trained longer enough, the resulting networks could try their best to exploit their capacity to reconstruct data (in other word, the networks could become less and less sparse). 
-- the author did not explain how the hyperparameters (of SPM, AIM and AOM) were chosen. The different hyperparameters could lead to very different conclusions. 
-- the sparseness measurement used in this paper is improper. The activations of sigmoid and tanh are unlike ReLU, at a certain scale. You could consider Hoyer’s sparseness measure (Hoyer 2004) which is on a normalized scale and avoids the hyperparameter epsilon in the SPM.
-- AOM should also consider the activation overlapping between samples from any number of classes'. 

Minor comments
Paragraph 4, page 2: The results of your DNNs are not state of the art for those data set.
Section 3: I cannot understand assumption 2.
In equation 2, what $x_{ij}$ represent?
In equation 3, m?