Nonparametric Weight Initialization of Neural Networks via Integral Representation
Authors: Sho Sonoda, Noboru Murata
ICLR 2014

Abstract: A new initialization method for hidden parameters in a neural network is proposed. Derived from the integral representation of the neural network, a nonparametric probability distribution of hidden parameters is introduced. In this proposal, hidden parameters are initialized by samples drawn from this distribution, and output parameters are fitted by ordinary linear regression. Numerical experiments show that backpropagation with proposed initialization converges faster than uniformly random initialization. Also it is shown that the proposed method achieves enough accuracy by itself without backpropagation in some cases.


Peer review:
[Review]:
This paper presents a new method for initializing the parameters of a feedforward neural network with a single hidden layer. The idea is to sample the parameters from a data-dependent distribution computed as an approximation of a kernel transformation of the target distribution. 

* The parameter initialization problem is important and the main idea of the paper is interesting. 

Now, computing the transformation of the target distribution is the same as solving an equation for the parameters of the network, analytically, assuming an unlimited number of hidden units. As this is a difficult problem, the method relies on an approximation of the parameter density and sampling therefrom N times when the actual network is assumed to have N hidden units. 

* It would be really helpful to have a notion of how expensive it is compute the approximation of the parameter density and to sample from it. Judging from the formulas this does not seem cheap. 

* The paper studies networks with sigmoid pairs. What can the authors say about sigmoid units? 

In Figure 1 left, the figure does not show that the support is non-convex, as claimed in the caption. 

The axes labels in Figure 1 are too small.