scikit-learn/scikit-learn — pull request #1395 by larsmans (2012-11-22 22:47:03+00:00)
Title: WIP multilayer perceptron classifier

This is a very early PR, just to let you know I'm working on this (and that I beat @amueller to it ;) and to keep a TODO list for myself.

This PR contains a working implementation of a multilayer perceptron classifier trained with batch gradient descent. Features:
* one hidden layer with logistic activation function
* momentum method (rolling average of gradients)
* multiclass cross-entropy loss
* [preliminary] 0.878 F1-score on small version of 20news (`n_hidden=400, alpha=0, max_iter=100, random_state=42, tol=1e-4, learning_rate=.02`)
* L2 penalty; doesn't seem to work very well, though, so turned off by default

TODO:
* switch to minibatch SGD
* reuse @pprett's Cython code for datasets and weight vectors
* reuse parts of `SGDClassifier`
* hinge loss
* multitarget/multilabel output
* evaluate different ways of computing matrix multiplications, which are the bottleneck in `fit` (so far, leveraging `np.dot` seems quite fast with a good BLAS)
* optimize further
* L1 penalty?
* Student's t penalty?
* [dropout](http://arxiv.org/pdf/1207.0580.pdf) regularization?
* [adaptive learning rates](http://arxiv.org/abs/1206.1106)?
* tanh hidden layer activation?
* tests
* demos
* documentation

Not TODO, at least not in this PR:
* regression
* conjugate gradient optimization
* multiple hidden layers