Feedforward nets with one hidden layer are closely related to projection pursuit regression. Probabilistic neural nets are identical to kernel discriminant analysis. General regression neural nets are identical to Nadaraya-Watson kernel regression. Kohonen nets for adaptive vector quantization are very similar to k-means cluster analysis. Hebbian learning is closely related to principal component analysis.
Feedforward nets are a subset of the class of nonlinear regression and discrimination models.
Statisticians have studied the properties of this general class but had not considered the specific case of feedforward neural nets before such networks were popularized in the neural network field. Still, many results from the statistical theory of nonlinear models apply directly to feedforward nets, and the methods that are commonly used for fitting nonlinear models, such as various Levenberg-Marquardt and conjugate gradient algorithms, can be used to train feedforward nets. While neural nets are often defined in terms of their algorithms or implementations, statistical methods are usually defined in terms of their results. What you get is still an arithmetic mean regardless of how you compute it. So a statistician would consider standard backprop, Quickprop, and Levenberg-Marquardt as different algorithms for implementing the same statistical model such as a feedforward net.