BVLC/caffe — pull request #330 by jeffdonahue (2014-04-16 05:08:26+00:00)
Title: MNIST autoencoder example

This PR moves `examples/lenet/` to `examples/mnist/` and adds the necessary files to train an autoencoder on MNIST with the architecture of Hinton & Salakhutdinov [1] using SGD with no pre-training (e.g. via RBM). It uses a sparse Gaussian initialization (added to `filler.hpp`) as suggested by [2] as a strategy for training autoencoders via SGD*. It uses a fixed LR of 0.0001 which could probably be greatly improved upon, but I haven't played with it much.

After 2 million iterations (which took a few hours on the GPU -- I didn't originally intend to train it this long but this is what it was at when I came back to it) the test L2 reconstruction error was around 1.5-1.6. (For an idea of what this means, a reconstruction with 2 out of 784 pixels flipped from perfectly white to perfectly black (or vice versa) would have an L2 reconstruction error of 2.0)

*actually [2] used Nesterov's accelerated gradient, but this is not currently implemented in Caffe and SGD seems to be fairly effective.

[1] http://www.cs.toronto.edu/~hinton/science.pdf

[2] http://jmlr.org/proceedings/papers/v28/sutskever13.pdf