RFC Unify old GradientBoosting estimators and HGBT

### Current situation
We have the unfortunate situation to have 2 different versions of gradient boosting, the old estimators ([`GradientBoostingClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html#sklearn-ensemble-gradientboostingclassifier) and `GradientBoostingRegressor`) as well as the new ones using binning and histogram strategies similar to LightGBM ([`HistGradientBoostingClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.HistGradientBoostingClassifier.html#sklearn-ensemble-histgradientboostingclassifier) and `HistGradientBoostingRegressor`).

This makes advertising the new ones harder, e.g. #26826, and also result in a larger feature gap between those two.
Based on discussions in #27139 and during a monthly meeting (maybe not documented), **I'd like to call for comments on the following:**

#### Proposition
Unify both types of gradient boosting in a single class, i.e. the old names `GradientBoostingClassifier` and make them switch the underlying estimator class based on a parameter value, e.g. `max_bins` (`None`->old classes, integer->new classes).

Note that binning and histograms are not the only difference.

### Comparison
#### Algorithm
The old GBT uses Friedman gradient boosting with a line search step. (The lines search sometimes, e.g. for log loss, uses a 2. order approximation and is therefore, sometimes, called "hybrid gradient-Newton boosting"). The trees are learned on the gradients. A tree searches for the best split among all (veeeery many) split candidates for all features. After a single tree is fit, the terminal node values are re-computed which corresponds to a line search step.

The new HGBT uses a 2. order approximation of the loss, i.e. gradients and hessians (XGBoost paper, therefore sometimes called Newton boosting). In addition, it bins/discretizes the features `X` and uses a histogram of gradients/hessians/counts per feature. A tree then searches for the best split candidate, but there are only n_features * n_bins candidates (muuuuch less than in GBT).

| estimator | trees train on        | node values (consequence of tree train) | features `X`   |
| --------- | --------------------- | ---------------------------- | -------------- |
| GBT       | gradients             | recomputed in lines search   | use as is      |
| HGBT      | gradients/hessians    | `sum(gradient)/sum(hessian)` | bin/discretize |

In fact, one could use 2. order loss (gradients and hessians) without binning `X`, and vice-versa, use binning with fitting trees on gradients (without hessians).

#### Parameters

| `HistGradientBoostingRegressor` | `GradientBoostingRegressor` | Same               | Comment |
| ------------------------------- | ----------------------------| ------------------ | ------- |
| loss                            | loss                        | :white_check_mark: | |
| quantile                        | alpha                       | :x:                | | 
| learning_rate                   | learning_rate               | :white_check_mark: | |
| max_iter                        | n_estimators                | :x:                | https://github.com/scikit-learn/scikit-learn/pull/12807#issue-391902947 |
| max_leaf_nodes                  | max_leaf_nodes              | :white_check_mark: |
| max_depth                       | max_depth                   | :white_check_mark: |
| min_samples_leaf                | min_samples_leaf            | :white_check_mark: |
| l2_regularization               | learning_rate               | :white_check_mark: |
| max_features                    | max_features                | :white_check_mark: |
| max_bins                        | :no_entry: (nonsense)       | :x:                |
| categorical_features            | :no_entry:                  | :x:                |
| monotonic_cst                   | :no_entry:                  | :x:                | #27305 |
| interaction_cst                 | :no_entry:                  | :x:                |
| warm_start                      | warm_start                  | :white_check_mark: |
| early_stopping                  | :no_entry:                  | :x:                |
| scoring                         | :no_entry:                  | :x:                |
| validation_fraction             | validation_fraction         | :white_check_mark: |
| n_iter_no_change                | n_iter_no_change            | :white_check_mark: |
| tol                             | tol                         | :white_check_mark: |
| verbose                         | verbose                     | :white_check_mark: |
| random_state                    | random_state                | :white_check_mark: |
| class_weight                    | :no_entry:                  | :x:                |
| :no_entry:                      | subsample                   | :x:                | #16062 |
| :no_entry: (nonsense)           | criterion                   | :x:                |
| :no_entry:                      | min_samples_split           | :x:                |
| :no_entry:                      | min_weight_fraction_leaf    | :x:                |
| :no_entry:                      | min_impurity_decrease       | :x:                |
| :no_entry:                      | init                        | :x:                | #27109 |
| :no_entry:                      | ccp_alpha                   | :x:                |

In fact, only the quantile/alpha and max_iter/n_estimator parameters are conflicting.

COMMENT:
I support the idea of making `HistGradientBoosting*` available as `GradientBoosting*`, and therefore the default. Our best "product" should be available at the prime location of our shop (aka name), not hidden under the counter. I think the proposed method of using the value of `n_bins` to choose between the two is the best idea so far for how to do this. But it does require significant effort and a few deprecation cycles :-/ c'est la vie.

I can see that by switching (massively?) the algorithm that you get based on a constructor argument you can create confusion. However, I wonder how many people who use gradient boosting actually care about the precise algorithm. I'd include people who can't explain the differences between the two in the category of "don't care" (this includes me).

---

Having both versions available under a dedicated name, say, `HistGradientBoosting*` and `ExactGradientBoosting*` (?) could be nice for people who want to be super explicit. You could take this idea further by offering more classes that use an estimator but hardwire some of the constructor arguments. The goal would be to make it simpler for users (dealing with the forest of constructor args) and increase discoverability. A downside would be a proliferation of class names. But this is going off-topic.

Labels: RFC