Talip7/scikit-learn-issues-multilabel huggingface dataset:
Allowing to group infrequent categories in `HistGradientBoosting`

### Describe the workflow you want to enable

`HistGradientBoostingClassifier` and `HistGradientBoostingRegressor` have built-in support for categorical features and use an `OrdinalEncoder` to encode them. Each feature must have less than `max_bins` (255) categories and if there are more we get a `ValueError`.

Would it be useful to add a parameter allowing to group together the least frequent categories when there are too many (and thus avoid the error)? This amounts to setting `max_categories=self.max_bins` in the internal `OrdinalEncoder`.

The workflow I would like to enable is fitting a HGB estimator when a categorical feature has a few too many categories, without needing to encode them myself beforehand.

### Describe your proposed solution

the HGB estimators would have a parameter (maybe something like `group_infrequent_categories`) to control whether they should display the current behavior (raise an error) or group together rare categories (with the OrdinalEncoder's `max_categories`) when there are more than `max_bins` categories

### Describe alternatives you've considered, if relevant

 It is relatively easy to apply an OrdinalEncoder before but (i) it is more verbose, and we have to use a pipeline and probably a ColumnTransformer, (ii) we lose the dtypes of pandas  dataframe columns so we have to specify the categorical columns manually instead of using `categorical_features="from_dtype"`, (iii) we end up doing the categorical feature encoding twice, once before and once inside the estimator

### Additional context

_No response_

COMMENT:
the encoders `OrdinalEncoder` and `OneHotEncoder` already implement this and it can be used by setting either `min_frequency` or `max_categories`, so all pipelines that deal with categories with those encoders can rely on it.
I'm not sure if that is what you are referring to.

besides the histogram gradient boosting, one place I see the OneHotEncoding being used is the `RandomTreesEmbedding` but in this case the number of categories is controlled by the number of trees. After a quick search I didn't find other estimators that use those encoders but there may be others that need to handle categories.

In the meanwhile the HistGradientBoosting already relies on OrdinalEncoder so the infrequent category handling is already built, AFAICT all that would be needed would be to take advantage of it by setting `OrdinalEncoder(max_categories=self.max_bins)`

Labels: New Feature
Allowing to group infrequent categories in `HistGradientBoosting`

### Describe the workflow you want to enable

`HistGradientBoostingClassifier` and `HistGradientBoostingRegressor` have built-in support for categorical features and use an `OrdinalEncoder` to encode them. Each feature must have less than `max_bins` (255) categories and if there are more we get a `ValueError`.

Would it be useful to add a parameter allowing to group together the least frequent categories when there are too many (and thus avoid the error)? This amounts to setting `max_categories=self.max_bins` in the internal `OrdinalEncoder`.

The workflow I would like to enable is fitting a HGB estimator when a categorical feature has a few too many categories, without needing to encode them myself beforehand.

### Describe your proposed solution

the HGB estimators would have a parameter (maybe something like `group_infrequent_categories`) to control whether they should display the current behavior (raise an error) or group together rare categories (with the OrdinalEncoder's `max_categories`) when there are more than `max_bins` categories

### Describe alternatives you've considered, if relevant

 It is relatively easy to apply an OrdinalEncoder before but (i) it is more verbose, and we have to use a pipeline and probably a ColumnTransformer, (ii) we lose the dtypes of pandas  dataframe columns so we have to specify the categorical columns manually instead of using `categorical_features="from_dtype"`, (iii) we end up doing the categorical feature encoding twice, once before and once inside the estimator

### Additional context

_No response_

COMMENT:
I have a PR up for this feature: https://github.com/scikit-learn/scikit-learn/pull/26268. API wise it adds a `on_high_cardinality_categories` to configure between grouping and erroring.

Labels: New Feature
Allowing to group infrequent categories in `HistGradientBoosting`

### Describe the workflow you want to enable

`HistGradientBoostingClassifier` and `HistGradientBoostingRegressor` have built-in support for categorical features and use an `OrdinalEncoder` to encode them. Each feature must have less than `max_bins` (255) categories and if there are more we get a `ValueError`.

Would it be useful to add a parameter allowing to group together the least frequent categories when there are too many (and thus avoid the error)? This amounts to setting `max_categories=self.max_bins` in the internal `OrdinalEncoder`.

The workflow I would like to enable is fitting a HGB estimator when a categorical feature has a few too many categories, without needing to encode them myself beforehand.

### Describe your proposed solution

the HGB estimators would have a parameter (maybe something like `group_infrequent_categories`) to control whether they should display the current behavior (raise an error) or group together rare categories (with the OrdinalEncoder's `max_categories`) when there are more than `max_bins` categories

### Describe alternatives you've considered, if relevant

 It is relatively easy to apply an OrdinalEncoder before but (i) it is more verbose, and we have to use a pipeline and probably a ColumnTransformer, (ii) we lose the dtypes of pandas  dataframe columns so we have to specify the categorical columns manually instead of using `categorical_features="from_dtype"`, (iii) we end up doing the categorical feature encoding twice, once before and once inside the estimator

### Additional context

_No response_

COMMENT:
great, sorry I missed it when searching for already existing issues about this!
(the link above has a typo, here it is: https://github.com/scikit-learn/scikit-learn/pull/26268)

Labels: New Feature
ENH support for missing values in ExtraTrees

### Describe the workflow you want to enable

Inspired  by https://github.com/scikit-learn/scikit-learn/pull/26391 I think that support for missing values for ExtraTrees regressor and classifier should/could also be provided.

### Describe your proposed solution

I think a foundational work is already provided by @thomasjpfan  in https://github.com/scikit-learn/scikit-learn/pull/26391 and besides tests and documentation to enable nan handling it is enough to modify `sklearn/tree/_classes.py`:
For `ExtraTreeRegressor` add method:
```
 def _more_tags(self):
        # XXX: nan is only support for dense arrays, but we set this for common test to
        # pass, specifically: check_estimators_nan_inf
        allow_nan = self.criterion in {
            "squared_error",
            "friedman_mse",
            "poisson",
        }
        return {"allow_nan": allow_nan}
```
For `ExtraTreeClassifier` add method:
```
def _more_tags(self):
        # XXX: nan is only support for dense arrays, but we set this for common test to
        # pass, specifically: check_estimators_nan_inf
        allow_nan = self.criterion in {
            "gini",
            "log_loss",
            "entropy",
        }
        return {"multilabel": True, "allow_nan": allow_nan}
```
I've run the code locally, and it appears to be functioning as expected. However, I must emphasize that my testing was not exhaustive, and I might have overlooked some obvious aspects.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
This is not as simple, I don't think that the random splitter is currently supporting the missing values, e.g.:

https://github.com/scikit-learn/scikit-learn/blob/0496f4877c2e6e7f050d573284001cc2ecdbbcf7/sklearn/tree/_splitter.pyx#L829-L832

So we need to change the splitter and not only the tags.

Labels: New Feature, module:tree
ENH support for missing values in ExtraTrees

### Describe the workflow you want to enable

Inspired  by https://github.com/scikit-learn/scikit-learn/pull/26391 I think that support for missing values for ExtraTrees regressor and classifier should/could also be provided.

### Describe your proposed solution

I think a foundational work is already provided by @thomasjpfan  in https://github.com/scikit-learn/scikit-learn/pull/26391 and besides tests and documentation to enable nan handling it is enough to modify `sklearn/tree/_classes.py`:
For `ExtraTreeRegressor` add method:
```
 def _more_tags(self):
        # XXX: nan is only support for dense arrays, but we set this for common test to
        # pass, specifically: check_estimators_nan_inf
        allow_nan = self.criterion in {
            "squared_error",
            "friedman_mse",
            "poisson",
        }
        return {"allow_nan": allow_nan}
```
For `ExtraTreeClassifier` add method:
```
def _more_tags(self):
        # XXX: nan is only support for dense arrays, but we set this for common test to
        # pass, specifically: check_estimators_nan_inf
        allow_nan = self.criterion in {
            "gini",
            "log_loss",
            "entropy",
        }
        return {"multilabel": True, "allow_nan": allow_nan}
```
I've run the code locally, and it appears to be functioning as expected. However, I must emphasize that my testing was not exhaustive, and I might have overlooked some obvious aspects.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Out of curiosity is there desire to work on adding the missing-value support to the random splitter? I am familiar w/ the codebase and would love to help move that along if there's interest from the core-devs. Or is @thomasjpfan already working on it?

Labels: New Feature, module:tree
ENH support for missing values in ExtraTrees

### Describe the workflow you want to enable

Inspired  by https://github.com/scikit-learn/scikit-learn/pull/26391 I think that support for missing values for ExtraTrees regressor and classifier should/could also be provided.

### Describe your proposed solution

I think a foundational work is already provided by @thomasjpfan  in https://github.com/scikit-learn/scikit-learn/pull/26391 and besides tests and documentation to enable nan handling it is enough to modify `sklearn/tree/_classes.py`:
For `ExtraTreeRegressor` add method:
```
 def _more_tags(self):
        # XXX: nan is only support for dense arrays, but we set this for common test to
        # pass, specifically: check_estimators_nan_inf
        allow_nan = self.criterion in {
            "squared_error",
            "friedman_mse",
            "poisson",
        }
        return {"allow_nan": allow_nan}
```
For `ExtraTreeClassifier` add method:
```
def _more_tags(self):
        # XXX: nan is only support for dense arrays, but we set this for common test to
        # pass, specifically: check_estimators_nan_inf
        allow_nan = self.criterion in {
            "gini",
            "log_loss",
            "entropy",
        }
        return {"multilabel": True, "allow_nan": allow_nan}
```
I've run the code locally, and it appears to be functioning as expected. However, I must emphasize that my testing was not exhaustive, and I might have overlooked some obvious aspects.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I think extending missing value to more tree based estimators is something that we'd like to see in the library. So a PR would be great! (There is of course always the caveat: someone has to have time to review it, and there might be other things that get reviewed first. But you've contributed to scikit-learn before so I think you know that :D)

Labels: New Feature, module:tree
ENH support for missing values in ExtraTrees

### Describe the workflow you want to enable

Inspired  by https://github.com/scikit-learn/scikit-learn/pull/26391 I think that support for missing values for ExtraTrees regressor and classifier should/could also be provided.

### Describe your proposed solution

I think a foundational work is already provided by @thomasjpfan  in https://github.com/scikit-learn/scikit-learn/pull/26391 and besides tests and documentation to enable nan handling it is enough to modify `sklearn/tree/_classes.py`:
For `ExtraTreeRegressor` add method:
```
 def _more_tags(self):
        # XXX: nan is only support for dense arrays, but we set this for common test to
        # pass, specifically: check_estimators_nan_inf
        allow_nan = self.criterion in {
            "squared_error",
            "friedman_mse",
            "poisson",
        }
        return {"allow_nan": allow_nan}
```
For `ExtraTreeClassifier` add method:
```
def _more_tags(self):
        # XXX: nan is only support for dense arrays, but we set this for common test to
        # pass, specifically: check_estimators_nan_inf
        allow_nan = self.criterion in {
            "gini",
            "log_loss",
            "entropy",
        }
        return {"multilabel": True, "allow_nan": allow_nan}
```
I've run the code locally, and it appears to be functioning as expected. However, I must emphasize that my testing was not exhaustive, and I might have overlooked some obvious aspects.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I'll be happy to review a PR for adding missing value support to the random splitter. I recommend starting with enabling missing value support for the decision tree and a follow up PR that adds ExtraTrees.

As with the original [missing values PR](https://github.com/scikit-learn/scikit-learn/pull/23595), we need to be careful and not introduce performance regressions.

There is one design decision to make that will influence the implementation: Do we want `DecisionTreeClassifier(splitter="random", random_state=0)` to give the same model with different scikit-learn versions?

Labels: New Feature, module:tree
ENH support for missing values in ExtraTrees

### Describe the workflow you want to enable

Inspired  by https://github.com/scikit-learn/scikit-learn/pull/26391 I think that support for missing values for ExtraTrees regressor and classifier should/could also be provided.

### Describe your proposed solution

I think a foundational work is already provided by @thomasjpfan  in https://github.com/scikit-learn/scikit-learn/pull/26391 and besides tests and documentation to enable nan handling it is enough to modify `sklearn/tree/_classes.py`:
For `ExtraTreeRegressor` add method:
```
 def _more_tags(self):
        # XXX: nan is only support for dense arrays, but we set this for common test to
        # pass, specifically: check_estimators_nan_inf
        allow_nan = self.criterion in {
            "squared_error",
            "friedman_mse",
            "poisson",
        }
        return {"allow_nan": allow_nan}
```
For `ExtraTreeClassifier` add method:
```
def _more_tags(self):
        # XXX: nan is only support for dense arrays, but we set this for common test to
        # pass, specifically: check_estimators_nan_inf
        allow_nan = self.criterion in {
            "gini",
            "log_loss",
            "entropy",
        }
        return {"multilabel": True, "allow_nan": allow_nan}
```
I've run the code locally, and it appears to be functioning as expected. However, I must emphasize that my testing was not exhaustive, and I might have overlooked some obvious aspects.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> There is one design decision to make that will influence the implementation: Do we want `DecisionTreeClassifier(splitter="random", random_state=0)` to give the same model with different scikit-learn versions?

My opinion is giving the same model sounds reasonable. I'm not too familiar w/ the intricacies here. Besides the random number generation, is there anything else we need to control?

Labels: New Feature, module:tree
ENH support for missing values in ExtraTrees

### Describe the workflow you want to enable

Inspired  by https://github.com/scikit-learn/scikit-learn/pull/26391 I think that support for missing values for ExtraTrees regressor and classifier should/could also be provided.

### Describe your proposed solution

I think a foundational work is already provided by @thomasjpfan  in https://github.com/scikit-learn/scikit-learn/pull/26391 and besides tests and documentation to enable nan handling it is enough to modify `sklearn/tree/_classes.py`:
For `ExtraTreeRegressor` add method:
```
 def _more_tags(self):
        # XXX: nan is only support for dense arrays, but we set this for common test to
        # pass, specifically: check_estimators_nan_inf
        allow_nan = self.criterion in {
            "squared_error",
            "friedman_mse",
            "poisson",
        }
        return {"allow_nan": allow_nan}
```
For `ExtraTreeClassifier` add method:
```
def _more_tags(self):
        # XXX: nan is only support for dense arrays, but we set this for common test to
        # pass, specifically: check_estimators_nan_inf
        allow_nan = self.criterion in {
            "gini",
            "log_loss",
            "entropy",
        }
        return {"multilabel": True, "allow_nan": allow_nan}
```
I've run the code locally, and it appears to be functioning as expected. However, I must emphasize that my testing was not exhaustive, and I might have overlooked some obvious aspects.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> Besides the random number generation, is there anything else we need to control?

The random number generation is the only thing to keep in mind.

> My opinion is giving the same model sounds reasonable. 

If we want to keep the same model, then I think the implementation becomes more complicated. We need to be very careful about drawing random values when considering the missing values. Any additional interaction with the original random state object will change the model. We likely need to create another rng object for any randomness used by missing value splitting.

Overall, I think it would be simpler to **not** have the same model. But it is a decision we need to make.

Labels: New Feature, module:tree
ENH support for missing values in ExtraTrees

### Describe the workflow you want to enable

Inspired  by https://github.com/scikit-learn/scikit-learn/pull/26391 I think that support for missing values for ExtraTrees regressor and classifier should/could also be provided.

### Describe your proposed solution

I think a foundational work is already provided by @thomasjpfan  in https://github.com/scikit-learn/scikit-learn/pull/26391 and besides tests and documentation to enable nan handling it is enough to modify `sklearn/tree/_classes.py`:
For `ExtraTreeRegressor` add method:
```
 def _more_tags(self):
        # XXX: nan is only support for dense arrays, but we set this for common test to
        # pass, specifically: check_estimators_nan_inf
        allow_nan = self.criterion in {
            "squared_error",
            "friedman_mse",
            "poisson",
        }
        return {"allow_nan": allow_nan}
```
For `ExtraTreeClassifier` add method:
```
def _more_tags(self):
        # XXX: nan is only support for dense arrays, but we set this for common test to
        # pass, specifically: check_estimators_nan_inf
        allow_nan = self.criterion in {
            "gini",
            "log_loss",
            "entropy",
        }
        return {"multilabel": True, "allow_nan": allow_nan}
```
I've run the code locally, and it appears to be functioning as expected. However, I must emphasize that my testing was not exhaustive, and I might have overlooked some obvious aspects.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I see. Maintaining a separate random number generator purely for backwards compatibility sake seems quite cumbersome. As a user, I would be okay having not having the same model then given this cost.

Labels: New Feature, module:tree
ENH support for missing values in ExtraTrees

### Describe the workflow you want to enable

Inspired  by https://github.com/scikit-learn/scikit-learn/pull/26391 I think that support for missing values for ExtraTrees regressor and classifier should/could also be provided.

### Describe your proposed solution

I think a foundational work is already provided by @thomasjpfan  in https://github.com/scikit-learn/scikit-learn/pull/26391 and besides tests and documentation to enable nan handling it is enough to modify `sklearn/tree/_classes.py`:
For `ExtraTreeRegressor` add method:
```
 def _more_tags(self):
        # XXX: nan is only support for dense arrays, but we set this for common test to
        # pass, specifically: check_estimators_nan_inf
        allow_nan = self.criterion in {
            "squared_error",
            "friedman_mse",
            "poisson",
        }
        return {"allow_nan": allow_nan}
```
For `ExtraTreeClassifier` add method:
```
def _more_tags(self):
        # XXX: nan is only support for dense arrays, but we set this for common test to
        # pass, specifically: check_estimators_nan_inf
        allow_nan = self.criterion in {
            "gini",
            "log_loss",
            "entropy",
        }
        return {"multilabel": True, "allow_nan": allow_nan}
```
I've run the code locally, and it appears to be functioning as expected. However, I must emphasize that my testing was not exhaustive, and I might have overlooked some obvious aspects.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@mglowacki100 perhaps you might be interested in taking a look at https://github.com/scikit-learn/scikit-learn/pull/27966 and seeing if it meets your interests?

@thomasjpfan I followed your workflow from the missing-value decisiontree PRs and think it might be ready for review if you end up having time.

Thanks!

Labels: New Feature, module:tree