scikit-learn/scikit-learn — Issue #543: Implement Average SGD
Author: ogrisel
Created: 2012-01-08 20:16:50+00:00
State: open

Average SGD is an old computational trick (namely Polyak Rupert averaging applied to SGD) that was recently rediscovered and theoretically studied by Wei Xu ([arxiv link](http://arxiv.org/abs/1107.2490)) and Francis Bach & Eric Moulines ([hal link](http://hal.archives-ouvertes.fr/hal-00608041/en)) and further empirically validated in the public implementation of SGD by Leon Bottou ([SGD project page](http://leon.bottou.org/projects/sgd)).

This trick makes the SGD algorithm converge much faster with a very small modification of the original SGD code. It would be great to have it implemented in the scikit along with some examples that reproduce Leon Bottou's experiments on the RCV1 and Alpha datasets (that would require 2 new dataset loaders).

@npinto also has a [pure python implementation](https://github.com/npinto/asgd) that could also serve as a reference for additional comparison benchmark.

@pprett the original author of the scikit-learn SGD module does not plan to work on this in the short term so anyone motivated enough, please feel free to embark (just send an email on the mailing to avoid // implementation efforts).