[WIP] Implementing Averaged SGD

Working on implementing averaged SGD for both classifier and regressor for #543
I'm curious what is the plan to expose this to the user. Probably as
separate AdaGradClassifier and AdaDeltaClassifier objects? In that case it
sounds good, otherwise I would fear that an attribute called
`step_size_function` might not be the place where I'd look for this.


On Thu, Jul 24, 2014 at 4:27 PM, Danny Sullivan <notifications@github.com>
wrote:

> In sklearn/linear_model/sgd_fast.pyx:
>
> > @@ -342,13 +366,17 @@ def plain_sgd(np.ndarray[double, ndim=1, mode='c'] weights,
> >                int learning_rate, double eta0,
> >                double power_t,
> >                double t=1.0,
> > -              double intercept_decay=1.0):
> > +              double intercept_decay=1.0,
> > +              bint avg=False):
>
> Alex and I were just talking and decided that because average is still not
> directly tied in to step_size we may end up with an implementation with 2
> parameters "average" and "step_size_function", where step_size_function is
> in {"adagrad", "adadelta", "optimal", ...}.
>
> —
> Reply to this email directly or view it on GitHub
> <https://github.com/scikit-learn/scikit-learn/pull/3480/files#r15349023>.
>