scikit-learn/scikit-learn — pull request #810 by mblondel (2012-05-02 10:41:23+00:00)
Title: MRG: New losses for SGD

- Renamed `p` to `epsilon`
- Allowed regression losses in `SGDClassifier` (the squared loss works surprisingly well if the learning rate is well set)
- Added the epsilon-insensitive loss (used by SVR)
- Added a sparse version of the log loss

I came up with this sparse log loss today. I changed the logistic function from `1 / (1 + e^-z)`, where `z = y * w^T x`, to `1 / (1 + e^-(az))` where `a` is chosen so that values of `z>=1-epsilon` are squashed to a probability of 1. Effectively, this results in sparse updates like the hinge loss but with probability support. Since I developed this loss just today, it may be too early to include it in scikit-learn. WDYT? Anyway, I tested it on the news20 dataset and it seems to work well (have a look at the density and training time).

```
================================================================================
log loss
________________________________________________________________________________
Training:
SGDClassifier(alpha=0.0001, class_weight=None, epsilon=0.1, eta0=0.0,
fit_intercept=True, learning_rate=optimal, loss=log, n_iter=50,
n_jobs=1, penalty=l2, power_t=0.5, rho=0.85, seed=0, shuffle=False,
verbose=0, warm_start=False)
train time: 0.369s
test time: 0.002s
f1-score: 0.896
dimensionality: 33807
density: 1.000000

================================================================================
sparse_log loss
________________________________________________________________________________
Training:
SGDClassifier(alpha=0.0001, class_weight=None, epsilon=0.1, eta0=0.0,
fit_intercept=True, learning_rate=optimal, loss=sparse_log,
n_iter=50, n_jobs=1, penalty=l2, power_t=0.5, rho=0.85, seed=0,
shuffle=False, verbose=0, warm_start=False)
train time: 0.255s
test time: 0.002s
f1-score: 0.899
dimensionality: 33807
density: 0.722668
```