scikit-learn/scikit-learn — pull request #3723 by staple (2014-09-30 05:53:36+00:00)
Title: [WIP] Choose out of bag scoring metric. Fixes #3455

Enhances the oob_score parameter of forest related predictors (e.g. RandomForestClassifier) to accept a string representing a scoring method or a callable scorer.

The existing out of bag scoring implementations use custom prediction logic in order to predict only out of bag input rows for each estimator. Additionally, per existing functionality the out of bag predictions are saved in documented fields of the predictor (oob_decision_function_, oob_prediction_). To simplify integration of this functionality with the existing scorer interface, the out of bag predictions are passed to a _DummyPredictor, which in turn makes them available to the scorer.

For backwards compatibility, the oob_score argument is not renamed, and it continues to accept True / False arguments (default still False). Passed True, the oob_score calculation is unchanged in the single output case (r2 for regression, accuracy for classification).

In the multi output case, the score calculations have changed. For regression, formerly individual r-squared metrics for each field were averaged across all output fields. Now a single r-squared metric is calculated across all outputs using the r2_score implementation. For classification, formerly the individual accuracy metrics were averaged across all output fields. Now a single accuracy metric is calculated based on complete accuracy of the set of outputs for a given input row, using the accuracy_score implementation.

This is a WIP, some known TODOS:
- Is the proposed implementation using DummyPredictor reasonable?
- Are the proposed scoring changes in the multi output case acceptable?
- Ensure that the accuracy scorer handles combined multi class + multi output cases
- Implement this for BaggingClassifier and BaggingRegressor as well?
- Tests, example etc