scikit-learn/scikit-learn — pull request #2198 by glouppe (2013-07-23 15:26:20+00:00)
Title: [WIP] Ensemble meta-estimator

Hi,

This is a very early PR for a meta-estimator ensemble implementing ensemble averaging/voting. The idea is to make a meta-estimator that can take as input any type of base estimator (not only trees) and make an ensemble out it.

Question about class names:
My first intention was to make a "Bagging" meta-estimator, ie an averaging ensemble which takes as input a base estimator and make an ensemble on bootstrap copies of the original dataset. However I find this arbitrarily limitating. I would like to:
- make drawing replacement an option (i.e., a boolean hyper-parameter `bootstrap`, like in forests)
- make it possible to chose the size of the subsamples the base estimators are built-on (i.e., a `subsampling` hyper-parameter, like in GBRT)
- then, I see no reason for not also making it possible to build base estimators on random subspaces of features. (Frustratingly, this dumb strategy works incredibly well in practice, and is often much more effective than classical bagging).

If all of these options are enabled, then I see no reason to call such an meta-estimator `BaggingClassifier` or `BaggingRegressor` (because this only applies for `bootstrap=True` + `subsampling=1.0`). Right now, I have named the classes `RandomPatchesClassifier` and `RandomPatchesRegressor`, as I did in my paper. However, I don't want to force my work to be pushed into scikit-learn.

Other suggestions I have in mind are `SubsamplingClassifier/Regressor` or something like that? What is your opinion about this? I'd like your input before proceeding.