scikit-learn/scikit-learn — pull request #448 by pprett (2011-11-24 08:36:58+00:00)
Title: Gradient boosting

This is a PR for Gradient Boosted Regression Trees [1] (aka Gradient Boosting, MART, TreeNet).

GBRTs have been advertised as one of the best off-the-shelf data-mining procedures; they share many properties with random forests including little need for tuning and data preprocessing as well as high predictive accuracy. GBRT have been used very successfully in areas such as learning of ranking functions and ecology.

This should be an alternative to R's 'gbm' package [2]. Currently, it feature three loss functions (binary classification, least squared regression and robust regression), stochastic gradient boosting, and variable importance.

The PR lacks documentation (there's only a stub in ensemble.rst), I'll update it ASAP.

I've benchmarked the code against R's gbm package (via rpy2) using a variety of datasets (about 4 classification and 3 regression datasets) - the results are remarkably similar; gbm, however, is usually a bit faster (I'll do some further profiling in the coming days).

Some features are still on my TODO list:

* Multi-class classification
* Partial dependency plots
* Quantile loss function for robust regression

I haven't benchmarked it against OpenCVs implementation - that's also on my TODO list.

[1] http://en.wikipedia.org/wiki/Gradient_boosting
[2] http://cran.r-project.org/web/packages/gbm/index.html