scikit-learn/scikit-learn — pull request #3883 by ogrisel (2014-11-25 17:01:50+00:00)
Title: [MRG] Improve stability of SGDClassifier / SGDRegressor with gradient clipping

The `squared_hinge` loss of `SGDClassifier` (and potentially the `squared` loss of `SGDRegressor`) tend to trigger numerical overflows depeinding even on well conditioned data for some hyper parameter combinations.

This PR fixes that issue by clipping `dloss` to 1e12. All other tests pass.

I have also add to prevent strong l2 regularization with large learning rates to trigger negative scales (which are meaningless). Instead I set the weights to zero in that case. The non regression tests highlight this case.

Both non regression tests were inspired by #3040. They both fail at epoch #2 and #3 of the iris data with the `sgd_fast.pyx` implementation from master.