lisa-lab/pylearn2 — pull request #420 by gdesjardins (2013-09-12 19:33:30+00:00)
Title: Momentum and AdaDelta as LearningRule

Added "learning_rule" parameter (object of type LearningRule) to SGD.__init__.

LearningRule are things which take as input: current parameter values, current gradients and a learning rate, and output the new parameter values. This mechanism can be used to implement standard SGD, SGD with momentum, AdaGrad, AdaDelta, and all other flavors of SGD.

Currently two LearningRules are implemented: Momentum and AdaDelta.

Yaml files and MomentumAdjustor have been updated to work with the new interface.

Old interface for momentum is still supported but raises a deprecated warning.

I've confirmed that I get the same results with the old and new interfaces using scripts/papers/maxout/mnist_pi.yaml (i.e. 1.14%). Note that this number differs from the published 0.94% result. This should be unrelated to the current pull request.