Why does the unsupervised pretraining encourage moderate-sparseness?
Authors: Jun Li, Wei Luo, Jian Yang, Xiaotong Yuan
ICLR 2014

Abstract: It is well known that direct training of deep multi-layer neural networks (DNNs) will generally lead to poor results. A major progress in recent years is the invention of various unsupervised pretraining methods to initialize network parameters and it was shown that such methods lead to good prediction performance. However, the reason for the success of the pretraining has not been fully understood, although it was argued that regularization and better optimization play certain roles. This paper provides another explanation for the effectiveness of the pretraining, where we empirically show the pretraining leads to a higher level of sparseness of hidden unit activation in the resulting neural networks, and the higher sparseness is positively correlated to faster training speed and better prediction accuracy. Moreover, we also show that rectified linear units (ReLU) can capture the sparseness benefits of the pretraining. Our implementation of DNNs with ReLU does not require the pretraining, but achieves comparable or better prediction performance than traditional DNNs with pretraining on standard benchmark datasets.


Peer review:
[Review]:
The authors study the effect of unsupervised pre-training on the levels of sparsity that hidden nodes have. The claimed contributions are that unsupervised pre-training encourages what they call “moderate-sparseness” and that their implementation of relus + fully-connected nets achieves state of the art performance on a number benchmarks such as MNIST, CIFAR-10 and JC-NORB.

I have found the beginning of Section 3.1 very confusing to read -- it requires some rephrasing (what is 1/R?) especially since it seems relatively crucial in terms of understanding the authors’ main point. The forward-references at the end of that Section are also confusing the reader. In Section 3.2 -- where does the 0.15 threshold come from? Why is there a jump in Figure 1? What happened at epoch 25?

It would be best if the authors put the best comparable numbers in the tables along with their *citations*, rather than just presenting their own results or other results without references attached.

It is unclear how the authors’ implementation of relus + fully-connected nets differs from the one initially proposed by Hinton and collaborators (as well as the others cited)? Why are the authors’ results better?

I’d say the paper is strangely structured/motivated: the claim that ReLUs produce sparser activations is believable and they measure it. The authors also measure that rectified units achieve better test errors (well-known before that). But I don’t see any evidence that sparsity of rectified units causes better generalization -- rectified units have other properties (such as the absence of saturation) that could be hypothesized to make them generalize better. I would say that the authors should work on finding a mechanism for why sparsity specifically is beneficial in supervised-backprop nets (the fact that unsupervised pre-training *also* leads to sparse representations, according to their metrics, is not a mechanism in and of itself). 

Generally the paper has a number of typos, awkward phrasings (e.g., “this paper does not intend to obliterate the contribution”)