scikit-learn/scikit-learn — pull request #2956 by y0ast (2014-03-10 21:12:21+00:00)
Title: Auto-encoder based on stochastic gradient Variational bayes

### Auto-encoder based on stochastic gradient Variational bayes

This is an auto-encoder based on Stochastic gradient Variational Bayes, a technique recently published by [Kingma and Welling](http://arxiv.org/abs/1312.6114). A month later a paper describing the same technique was published by [Rezende et al.](http://arxiv.org/abs/1401.4082) at Deepmind (the deep learning start-up recently acquired by Google). It's a great improvement on the Wake Sleep algorithm, but for more info I refer to the papers which describe it more eloquently than I could.

I developed this code together with Otto Fabius as part of a project for our MSc AI at the University of Amsterdam. This project was under the supervision of PhD candidate Diederik Kingma, who had the original idea and wrote the paper. It was originally written with the Theano dependency, for this pull request I rewrote the code to include the manually derived gradients and get rid of the Theano dependency.

It supports both continuous and discrete data (for more info refer to the paper by Kingma and Welling) and I think most requirements listed on the website are met. I tried to follow the implementation of the RBM and some code will probably be familiar :smiley: .

Most important to do is to include tests for continuous data, but I was unsure on what data I should base this. Would Olivetti faces be a good idea? Also the code is not tested on Python 3, this should be easy (change the prints).

Otto wrote a demo which compares logistic regression results on the raw pixel data, the data transformed by an RBM and the data transformed by this auto-encoder. It relies on MNIST data provided on the deeplearning website, which should probably be changed to data already incorporated in sci-kit learn. It beats the RBM, although the auto-encoder has to train much longer.

This is my first pull request, so if I did anything odd that's because my learning hasn't converged yet :wink: . I am happy to make corrections and incorporate suggestions where necessary :smiley: .

I will now also send an email to the mailing-list for easier communication.