Talip7/scikit-learn-issues-multilabel huggingface dataset:
RFC Move `_more_tags` to "developer API" via `__sklearn_tags__`

As a part of making it easier and more "standard" to write scikit-learn estimators by third party developers, we have been slowly developing a "developer API" kind of thing, which are useful for third party developers, but not end users of the estimators.

Some of the work has been:
- `__sklearn_clone__`
- `__metadata_request__fit`, ...
- `get_metadata_routing`

What I'm proposing here, is to create a new `__sklearn__tags__` method instead of the existing `_more_tags`.

We've had a lot of discussions when we designed the current system, which goes through the MRO and the `_more_tags` _adds_ to the tag set instead of returning the tags. Now the question is do we want to keep the current system or do we want `__sklearn_tags__` to return the estimator's tags instead, and call parent's `__sklearn_tags__` inside itself? As in, instead of:

```py
class Estimator(BaseEstimator):
    ...
    def _more_tags(self):
        return {...}
```

to have:

```py
class Estimator(BaseEstimator):
    ...
    def __sklearn_tags__(self):
        tags = super().__sklearn_tags__()
        # update tags
        return return tags
```
Inside our tags, we also have some starting with `_` such as `_xfail`, and the question is do we want to make those _public_.

cc @scikit-learn/core-devs

COMMENT:
I agree, before merging #22606 I think it'd be nice to clean up our tags. I'll work on it and open PRs.

Labels: API, RFC, Developer API
RFC Move `_more_tags` to "developer API" via `__sklearn_tags__`

As a part of making it easier and more "standard" to write scikit-learn estimators by third party developers, we have been slowly developing a "developer API" kind of thing, which are useful for third party developers, but not end users of the estimators.

Some of the work has been:
- `__sklearn_clone__`
- `__metadata_request__fit`, ...
- `get_metadata_routing`

What I'm proposing here, is to create a new `__sklearn__tags__` method instead of the existing `_more_tags`.

We've had a lot of discussions when we designed the current system, which goes through the MRO and the `_more_tags` _adds_ to the tag set instead of returning the tags. Now the question is do we want to keep the current system or do we want `__sklearn_tags__` to return the estimator's tags instead, and call parent's `__sklearn_tags__` inside itself? As in, instead of:

```py
class Estimator(BaseEstimator):
    ...
    def _more_tags(self):
        return {...}
```

to have:

```py
class Estimator(BaseEstimator):
    ...
    def __sklearn_tags__(self):
        tags = super().__sklearn_tags__()
        # update tags
        return return tags
```
Inside our tags, we also have some starting with `_` such as `_xfail`, and the question is do we want to make those _public_.

cc @scikit-learn/core-devs

COMMENT:
Done in https://github.com/scikit-learn/scikit-learn/pull/29677

Labels: API, RFC, Developer API
Parameter Validation Documentation?

While implementing a custom estimator, I noticed that the BaseEstimator class brings in a `_validate_params` method. Looking through this repo's history, it looks like it came in back during 2022 as part of PR https://github.com/scikit-learn/scikit-learn/pull/22722

```python
    def _validate_params(self):
        """Validate types and values of constructor parameters

        The expected type and values must be defined in the `_parameter_constraints`
        class attribute, which is a dictionary `param_name: list of constraints`. See
        the docstring of `validate_parameter_constraints` for a description of the
        accepted constraints.
        """
        validate_parameter_constraints(
            self._parameter_constraints,
            self.get_params(deep=False),
            caller_name=self.__class__.__name__,
        )
```

Beyond the PR itself and a docstring in `utils._param_validation.py` there does not seem to be much information about this method. The string "_validate_params" returns no results on the [web documentation](https://scikit-learn.org/stable/search.html?q=_validate_params). The [Developing Scikit-Learn Estimators](https://scikit-learn.org/stable/developers/develop.html) documentation also does not mention this tooling. so the only way to learn how to use it is to poke through the source code. 

Looking around further, it seems like in the time since that PR, most of the estimators in the package now use the `_fit_context` decorator defined in `base.py` which indirectly calls the `_validate_params` method. That decorator also [never appears](https://scikit-learn.org/stable/search.html?q=_fit_context) in the web documentation.

For those of us who develop custom estimators that extend Sklearn's base classes, it is useful to re-use the tooling that already exists (especially when that tooling comes from Sklearn itself). I am curious about a few things I had trouble finding answers to:
- Is there documentation on the canonical way to handle parameter validation?
- This version of parameter validation relies heavily on definitions in `utils._param_validation`; because this is prefixed with an underscore, is it "safe" to import this as a dependency in downstream packages?
- Is there any intended / planned relationship between `utils._param_validation` and `utils.validation`?

COMMENT:
Hi @jeremiedbb

Thank you for the quick response, and for answering my questions! Looking forward to developer API you mentioned.

Labels: Documentation
Automatically handle missing values in OrdinalEncoder

### Describe the workflow you want to enable

Currently, NaN values in OrdinalEncoder are either passed through as NaN, or encoded into user-specified value.

It would be nice to have a third option: consider NaN values as another category and map them into `num_categories + 1` or some other value.



### Describe your proposed solution

Add another `encoded_missing_value` option `auto`, that encodes them into another category

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

There is also some confusion with user-specified value: if for example I set this value as `0`, will it interfere with `0` category that was present during fit? Or all categories will be moved accordingly? 

Manually setting some other values like `1000000` or `-1` is usually incompatible with common categorical features interfaces, e.g. nn.Embedding from Pytorch and so on

COMMENT:
I think that the request is well motivated. We did not think about integration with `torch.nn.Embedding` when suggesting to use `-1` as a fixed category in our docstring example.

I would be +1 for a PR to implement the `"auto"` strategy described above (encode non-missing from `0` to `num_categories - 1` and use `num_categories` for missing.

However I suppose we would need to do something similar for `unknown_value` (when `handle_unknown="use_encoded_value"`).

Any opinion on how to handle hparam interactions with a sane naming scheme?

/cc @thomasjpfan

Labels: New Feature, API, Needs Decision, module:preprocessing
Automatically handle missing values in OrdinalEncoder

### Describe the workflow you want to enable

Currently, NaN values in OrdinalEncoder are either passed through as NaN, or encoded into user-specified value.

It would be nice to have a third option: consider NaN values as another category and map them into `num_categories + 1` or some other value.



### Describe your proposed solution

Add another `encoded_missing_value` option `auto`, that encodes them into another category

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

There is also some confusion with user-specified value: if for example I set this value as `0`, will it interfere with `0` category that was present during fit? Or all categories will be moved accordingly? 

Manually setting some other values like `1000000` or `-1` is usually incompatible with common categorical features interfaces, e.g. nn.Embedding from Pytorch and so on

COMMENT:
> I would be +1 for a PR to implement the "auto" strategy described above (encode non-missing from 0 to num_categories - 1 and use num_categories for missing.

I am +1 on the strategy. I would give it a more descriptive name such as: `encoded_missing_value="num_categories"` or `"cardinality"`. 

> However I suppose we would need to do something similar for unknown_value

Are you thinking of encoding unknown values as `num_categories` as well?

Labels: New Feature, API, Needs Decision, module:preprocessing
Automatically handle missing values in OrdinalEncoder

### Describe the workflow you want to enable

Currently, NaN values in OrdinalEncoder are either passed through as NaN, or encoded into user-specified value.

It would be nice to have a third option: consider NaN values as another category and map them into `num_categories + 1` or some other value.



### Describe your proposed solution

Add another `encoded_missing_value` option `auto`, that encodes them into another category

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

There is also some confusion with user-specified value: if for example I set this value as `0`, will it interfere with `0` category that was present during fit? Or all categories will be moved accordingly? 

Manually setting some other values like `1000000` or `-1` is usually incompatible with common categorical features interfaces, e.g. nn.Embedding from Pytorch and so on

COMMENT:
@thomasjpfan @ogrisel 

> Are you thinking of encoding unknown values as num_categories as well?

I would advise against that, or at least make it configurable. In many cases, absense of a feature (i.e. nan) has a lot of signal (for example, it might mean "no data from certain source"), and this signal is different from "has some data, but it hasn't been seen before"

Another option would be to encode rare categories as most common category - so we have the option to either automatically map to most common, or map into user-set category.

Labels: New Feature, API, Needs Decision, module:preprocessing
Automatically handle missing values in OrdinalEncoder

### Describe the workflow you want to enable

Currently, NaN values in OrdinalEncoder are either passed through as NaN, or encoded into user-specified value.

It would be nice to have a third option: consider NaN values as another category and map them into `num_categories + 1` or some other value.



### Describe your proposed solution

Add another `encoded_missing_value` option `auto`, that encodes them into another category

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

There is also some confusion with user-specified value: if for example I set this value as `0`, will it interfere with `0` category that was present during fit? Or all categories will be moved accordingly? 

Manually setting some other values like `1000000` or `-1` is usually incompatible with common categorical features interfaces, e.g. nn.Embedding from Pytorch and so on

COMMENT:
I agree with this.  I think it would be nice to map missing to 0 and unknown to 1 and shift the rest of the categories to 2 ... n+2.

And then have an attribute that tells you the max encoding!  So you don't have to save that info elsewhere or wrap OrdinalEncoder in some custom class.  This is important because that max integer is input into the entity embeddings, and so having that info readily available is very handy

I have a blog post on issues with unseen levels in categoricals.  IMO mapping to the most common category is risky.  Unseen levels tend to be overfit also unless you do data augmentation: https://medium.com/data-science/data-disruptions-to-elevate-entity-embeddings-b1ddf86a3c95

Labels: New Feature, API, Needs Decision, module:preprocessing
Easily retrieve mapping from OrdinalEncoder

### Describe the workflow you want to enable

It would be nice to be able to easily retrieve mapping in the form of a dictionary
```
"category_a": 0,
"category_b": 1,
"category_infrequent": 2,
...
```

Currently .categories_ attribute only retrieves list of seen categories, without mapping.

This becomes especially important with options to handle missing or infrequent values, which are leading to questions "What value does infrequent categories map to?" and so on.

### Describe your proposed solution

Add .categories_map_ attribute to OrdinalEncoder

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
It's possible to construct the category maps via list and dict comprehensions:

```python
category_maps = [
    {cat_name: cat_idx for cat_idx, cat_name in enumerate(cat_list)}
    for cat_list in oe.categories_
]
```

or as a dict of dicts to leverage feature names when available:

```python
category_maps = {
    feature_name: {cat_name: cat_idx for cat_idx, cat_name in enumerate(cat_list)}
    for feature_name, cat_list in zip(oe.feature_names_in_, oe.categories_)
}
```

Maybe we could expose a public property (or two, in the presence of the feature names) on the `OrdinalEncoder` but I also worry about the magic aspect of properties.

Precomputing those reverse mappings is also possible but redundant.

Another alternative would to use methods (instead of properties) to generate those reverse mappings on depend, maybe with optional argument to do it only for a specific set of features.

Labels: New Feature, API, Needs Decision
Easily retrieve mapping from OrdinalEncoder

### Describe the workflow you want to enable

It would be nice to be able to easily retrieve mapping in the form of a dictionary
```
"category_a": 0,
"category_b": 1,
"category_infrequent": 2,
...
```

Currently .categories_ attribute only retrieves list of seen categories, without mapping.

This becomes especially important with options to handle missing or infrequent values, which are leading to questions "What value does infrequent categories map to?" and so on.

### Describe your proposed solution

Add .categories_map_ attribute to OrdinalEncoder

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> This becomes especially important with options to handle missing or infrequent values, which are leading to questions "What value does infrequent categories map to?" and so on.

I am not sure how providing encoding maps would help answer that question: what would be the "key" in the dict for infrequent or unseen categories?

But we could expose dedicated fitted attributes to make the resulting encoding of missing and infrequent/unseen categories explicit for each feature.

EDIT: such an attribute already exists for infrequent categories: [OrdinalEncoder.infrequent_categories_](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OrdinalEncoder.html)

Labels: New Feature, API, Needs Decision
Easily retrieve mapping from OrdinalEncoder

### Describe the workflow you want to enable

It would be nice to be able to easily retrieve mapping in the form of a dictionary
```
"category_a": 0,
"category_b": 1,
"category_infrequent": 2,
...
```

Currently .categories_ attribute only retrieves list of seen categories, without mapping.

This becomes especially important with options to handle missing or infrequent values, which are leading to questions "What value does infrequent categories map to?" and so on.

### Describe your proposed solution

Add .categories_map_ attribute to OrdinalEncoder

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@ogrisel 

Thank you for your comments!

I will try to argue more about my motivation for each question:
1. Possible to construct the mapping by hand - my main gripe with this approach is that the assumption "order of categories in this list == encoded labels" is currently not outlined in the docs. So the solution would be to either outline encoding algorithm (inc. missing/unseen values) it in the docs, or to give method to produce exact mapping (which would still be very useful in reducing boilerplate code in downstream applications)

2.1. What would be the "key" for unseen categories - I think it is either some predefined "_unseen_category_hope_not_in_user_data", which is ugly, or possibly some new attribute like .unseen_encoded_value

2.2. OrdinalEncoder.infrequent_categories_ - while it is very useful for checking what categories turned out to be infrequent, it still doesnt answer the question into which value they are being encoded.

Labels: New Feature, API, Needs Decision
Add missing value support to ExtraTreesRegressor

### Describe the workflow you want to enable

It wasn't very clear to me from the version 1.4 release notes and I inferred that missing value support was added for all DecisionTreeRegressor based regressors. I've noticed though that the `ExtraTreesRegressor` does not support missing values the same as `RandomForestRegressor` does. The documentation page even mentions that monotonicity constraints are not supported for "regressions trained on data with missing values." - even though that is apparently not possible (there is an exception that clearly states that it does not when trying it).

### Describe your proposed solution

No idea about the details of a possible solution but it would be nice if this feature is also added for the `ExtraTreesRegressor`

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I think there is no fundamental reason why `ExtraTreesRegressor` and `ExtraTreesClassifier` don't support missing values. If someone wants to work on adding this, or the first step of it whic his figuring out what needs doing that would be great.

Labels: New Feature
Add missing value support to ExtraTreesRegressor

### Describe the workflow you want to enable

It wasn't very clear to me from the version 1.4 release notes and I inferred that missing value support was added for all DecisionTreeRegressor based regressors. I've noticed though that the `ExtraTreesRegressor` does not support missing values the same as `RandomForestRegressor` does. The documentation page even mentions that monotonicity constraints are not supported for "regressions trained on data with missing values." - even though that is apparently not possible (there is an exception that clearly states that it does not when trying it).

### Describe your proposed solution

No idea about the details of a possible solution but it would be nice if this feature is also added for the `ExtraTreesRegressor`

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Duplicate of https://github.com/scikit-learn/scikit-learn/issues/27931

Labels: New Feature
Add missing value support to ExtraTreesRegressor

### Describe the workflow you want to enable

It wasn't very clear to me from the version 1.4 release notes and I inferred that missing value support was added for all DecisionTreeRegressor based regressors. I've noticed though that the `ExtraTreesRegressor` does not support missing values the same as `RandomForestRegressor` does. The documentation page even mentions that monotonicity constraints are not supported for "regressions trained on data with missing values." - even though that is apparently not possible (there is an exception that clearly states that it does not when trying it).

### Describe your proposed solution

No idea about the details of a possible solution but it would be nice if this feature is also added for the `ExtraTreesRegressor`

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I had been looking for that issue but searching for ExtraTreesRegressor/ExtraTreesClassifier didn't find it. Yay

Labels: New Feature
Configure OpenBLAS to use scikit-learn's OpenMP threadpool

OpenBLAS v0.3.28 will have a new feature allowing OpenBLAS to use the threadpool chosen by the user, (see https://github.com/OpenMathLib/OpenBLAS/pull/4577).

This is very interesting because it would solve a performance issue happening when there's a quick succession of BLAS calls and OpenMP (prange) calls. The issue happens when OpenBLAS and OpenMP don't share the same threadpool because both threadpools are in active wait mode when they're idle (see https://github.com/OpenMathLib/OpenBLAS/issues/3187 for details), which is a current situation since numpy and scipy wheels are built against OpenBLAS with the pthreads threading layer.

This issue is currently impacting some estimators like KMeans (https://github.com/scikit-learn/scikit-learn/issues/20642), NMF (https://github.com/scikit-learn/scikit-learn/pull/16439), pairwise_distances (https://github.com/scikit-learn/scikit-learn/issues/26097), ...

Being able to configure OpenBLAS to use our OpenMP threadpool would allow to get rid of this issue even if numpy and scipy keep building their wheels against OpenBLAS pthreads (which is very likely).

I'm not sure yet if or how https://github.com/OpenMathLib/OpenBLAS/pull/4577 would make this possible so I'm opening this issue to track the progress on this subject.

COMMENT:
This is indeed interesting and might also be of interest for other people in the ecosystem, e.g. @rgommers and numpy/scipy developers interested in multithreading.

Labels: Performance
Configure OpenBLAS to use scikit-learn's OpenMP threadpool

OpenBLAS v0.3.28 will have a new feature allowing OpenBLAS to use the threadpool chosen by the user, (see https://github.com/OpenMathLib/OpenBLAS/pull/4577).

This is very interesting because it would solve a performance issue happening when there's a quick succession of BLAS calls and OpenMP (prange) calls. The issue happens when OpenBLAS and OpenMP don't share the same threadpool because both threadpools are in active wait mode when they're idle (see https://github.com/OpenMathLib/OpenBLAS/issues/3187 for details), which is a current situation since numpy and scipy wheels are built against OpenBLAS with the pthreads threading layer.

This issue is currently impacting some estimators like KMeans (https://github.com/scikit-learn/scikit-learn/issues/20642), NMF (https://github.com/scikit-learn/scikit-learn/pull/16439), pairwise_distances (https://github.com/scikit-learn/scikit-learn/issues/26097), ...

Being able to configure OpenBLAS to use our OpenMP threadpool would allow to get rid of this issue even if numpy and scipy keep building their wheels against OpenBLAS pthreads (which is very likely).

I'm not sure yet if or how https://github.com/OpenMathLib/OpenBLAS/pull/4577 would make this possible so I'm opening this issue to track the progress on this subject.

COMMENT:
Wow, this is amazing! From looking through, https://github.com/OpenMathLib/OpenBLAS/pull/4577, I think we'll need to write and register a callback that hooks up scikit-learn's vendered OpenMP with OpenBLAS.

If this backend-specific callback code is useful for other projects, is there a way to share it through `threadpoolctl`? Concretely, something like `threadpoolctl.register_openblas_backend("openmp")`. Although, this does increase the scope of `threadpoolctl`, the feature feels related.

Labels: Performance