scikit-learn/scikit-learn — pull request #2120 by IssamLaradji (2013-06-30 21:35:06+00:00)
Title: (WIP) Adding Multi-layer perceptron (MLP) to neural_network folder

[WIP] Multi-layer perceptron (MLP)

![mlp](https://f.cloud.github.com/assets/3382128/727899/5112c4dc-e1cb-11e2-8eaa-fe9b87bd9724.jpg)

A multilayer perceptron (MLP) is a feedforward artificial neural network model that tries to learn a function f(X)=y where y is the output and X is the input. An MLP consists of multiple layers, usually of one hidden layer, an input layer and an output layer, where each layer is fully connected to the next one. This is a classic algorithm that has been extensively used in Neural Networks.

Properties:

- Implemented naive backpropagation (that doesn't involve a fancy scipy's optimizer such as l_bfgs nor conjugate gradient (cg)) algorithm, that supports,
- partial_fit (online updating)
- mini batch processing
- Added scipy's optimizer 'fmin_l_bfgs_bf' as a second option for optimizing the weights
- It is very efficient and powerful (see Benchmark results below).
- The code abides to flake8 python convention
- Except the rules that involve lowercasing parameters (sometimes uppercase letters are more readable as variables)
- Supports sparse matrices
- The gradient was tested by computing it numerically and analytically
- Everything is vectorized
- The algorithm was developed based on the tutorials/references given by Prof. Andrew NG,
- [1] http://ufldl.stanford.edu/wiki/index.php/Backpropagation_Algorithm

TODO:

- Fix non-standard code to adhere to scikit's convention
(some practices used in the code may not be efficient nor reader friendly)
- Add test cases
- Add scipy's 'fmin_cg' as a third option for optimizing the weights
- Add a complete example on using MLP
- Add more loss functions
- Add more activation functions
- Remove the comments and debugging functions from the mlp.py file
- They are there for debugging purposes only

Benchmarks:

- Compared the performance of Naive backpropagation (NBp) and scipy's optimizer 'fmin_l_bfgs_bf' (l_bfgs) on the popular XML classification dataset (Sigmod) (only text was modeled using scikit's tf-idf)
- Via 100 iterations while other parameters are kept the same,
- Average score of NBp is 0.93
- Average score of l_bfgs is 0.98
- They both took around the same time

This is an extention to amueller and larsmans code on MLP.