Nonparametric Weight Initialization of Neural Networks via Integral Representation
Authors: Sho Sonoda, Noboru Murata
ICLR 2014

Abstract: A new initialization method for hidden parameters in a neural network is proposed. Derived from the integral representation of the neural network, a nonparametric probability distribution of hidden parameters is introduced. In this proposal, hidden parameters are initialized by samples drawn from this distribution, and output parameters are fitted by ordinary linear regression. Numerical experiments show that backpropagation with proposed initialization converges faster than uniformly random initialization. Also it is shown that the proposed method achieves enough accuracy by itself without backpropagation in some cases.


Peer review:
[Review]:
This paper introduces a new method for initializing the weights of a neural network. The technique is based on integral transforms. The function to learn f is represented as an infinite combination of basis functions weighted by some distribution. Conversely, this distribution can be obtained by projecting the function f onto another (related) set of basis functions evaluated at every point x in the input space.

The powerful analytic framework yields a probability distribution from which initial parameters of the neural network can be sampled. This is done using an acceptance-rejection sampling method. In order to overcome the computational inefficiency of the basic procedure, the authors propose a coordinate transform method that reduces the rejection rate. It would be useful to have more information on the order of magnitude by which the method is slower/faster compared to training a classically initialized neural network, and how does the method scale with the number of data points and the dimensions of the input space.

The experimental section consists of three experiments measuring the convergence of learning for various datasets (two low-dimensional toy examples and MNIST). On the low-dimensional toy examples, the proposed initialization is shown to be superior to uniform. However, these two datasets are to a certain extent already well modeled by local methods, for which good initialization heuristics are readily available (e.g. RBF networks + k-means). I am concerned about the validity of the MNIST experiment where a baseline error of >0.8 (80%?) is obtained with 1000 samples while other papers typically report 10% error for similar amount of data.