scikit-learn/scikit-learn — pull request #2123 by larsmans (2013-07-01 16:08:10+00:00)
Title: WIP new, simpler scorer API

Here's a simpler scorer API. Sorry for being so late to the game, I was too busy to bother with the discussions. I hope no-one takes offense (esp. @amueller who worked hard on this). Ping @jnothman. Anyway, the highlights:

* Scorers are just callables that take `estimator, X, y, **kwargs`.
* No more `greater_is_better`. A scorer made from a loss function returns minus the loss.
* No more `Scorer` class. There's a factory function `make_scorer`, but it produces objects of an internal class hierarchy that do their best to hide as callables.
* There's a currently unused scorer class for probabilistic classification, in anticipation of #2013.

TODO:
* Docs needs copy-editing, see commit message.
* Maybe scorers always need to return a tuple; the first element is then the score, the rest is recorded and reported by `GridSearchCV` and friends. That would solve #1850.
* Should `make_scorer` be public at all?