Nonparametric Weight Initialization of Neural Networks via Integral Representation
Authors: Sho Sonoda, Noboru Murata
ICLR 2014

Abstract: A new initialization method for hidden parameters in a neural network is proposed. Derived from the integral representation of the neural network, a nonparametric probability distribution of hidden parameters is introduced. In this proposal, hidden parameters are initialized by samples drawn from this distribution, and output parameters are fitted by ordinary linear regression. Numerical experiments show that backpropagation with proposed initialization converges faster than uniformly random initialization. Also it is shown that the proposed method achieves enough accuracy by itself without backpropagation in some cases.


Peer review:
[Review]:
The paper proposes a new pre-training scheme for neural networks which are to be fine-tuned later with back propagation.

This pre-training scheme is done in two steps
 1 - sampling the parameters of the first layer using Importance sampling or an accept-reject MCMC method (both methods  are apparently confused by the authors) in a data dependent way.
 2 - train the parameters of the output layer using linear regression.

The experiments compare the test RMSE/error-rate obtained using traditional back propagation and that obtained using the proposed method, on three datasets: a 1D function, a Boolean function and the Mnist dataset.

The proposed pre-training scheme is new but the scientific quality of the paper is questionable. First the proposed method is given a misleading name since it proposes to do the initialization in a data dependent way (with a linear regression step). This may be understood as a 'pre-training scheme', not as an 'initialization'. Second, the paper is very misleading in its report of previous work, for instance stating that (Efficient Backprop, Le Cun 1998) proposes to initialize neural networks by sampling from a uniform distribution [-1/sqrt(fan-in);1/sqrt(fan-in)] when it suggests in fact to sample from a normal distribution of mean zero and standard deviation sigma=1/sqrt(fan-in).

Additionally, the experiments are again very misleading. First, the main claim of the paper is that using the proposed pre-training scheme, BP will converge faster. However, the time to convergence is reported in terms of the number of BP iterations and does not take the pre-training time into account. This is especially worrisome since the pre-training scheme relies on MCMC sampling which is usually very computationally expensive compared to back propagation. Finally, the results reported on the Mnist dataset are inconsistent with previous work when then give a test error rate for back-propagation and 300 hidden units around 90% when it should be around 1.6% (cf. Mnist dataset website). 

pros:
cons:
 - Misleading summary of previous work.
 - Misleading reference to an initialization strategy which is in fact a data dependent pre-training step.
 - Experiments do not report the pre-training time and are therefore strongly biased in favor of the proposed method.
 - Results on Mnist are inconsistent with previous work.