Talip7/scikit-learn-issues-multilabel huggingface dataset:
HDBSCAN error with metric cosine

### Describe the bug

Inconsistent HDBSCAN behavior when given a metric that is not supported by KDTree or BallTree.

[docs](https://scikit-learn.org/stable/modules/generated/sklearn.cluster.HDBSCAN.html)

```
metric : str or callable, default=’euclidean’

    The metric to use when calculating distance between instances in a feature array.

        If metric is a string or callable, it must be one of the options allowed by [pairwise_distances](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.pairwise_distances.html#sklearn.metrics.pairwise_distances) for its metric parameter.

        If metric is “precomputed”, X is assumed to be a distance matrix and must be square.
```

### Steps/Code to Reproduce

```py
from sklearn.cluster import HDBSCAN

clusterer = HDBSCAN(metric="cosine")
clusterer.fit([])
```

### Expected Results

no error

### Actual Results

InvalidParameterError: The 'metric' parameter of HDBSCAN must be a str among {'euclidean', 'p', 'rogerstanimoto', 'seuclidean', 'l1', 'l2', 'russellrao', 'cityblock', 'sokalmichener', 'precomputed', 'dice', 'manhattan', 'minkowski', 'pyfunc', 'jaccard', 'chebyshev', 'infinity', 'mahalanobis', 'hamming', 'braycurtis', 'haversine', 'canberra', 'sokalsneath'} or a callable. Got 'cosine' instead.

### Versions

```shell
System:
    python: 3.11.7 (main, Dec  4 2023, 18:10:11) [Clang 15.0.0 (clang-1500.1.0.2.5)]
executable: /opt/homebrew/opt/python@3.11/bin/python3.11
   machine: macOS-14.3.1-arm64-arm-64bit

Python dependencies:
      sklearn: 1.4.1.post1
          pip: 24.0
   setuptools: 69.0.2
        numpy: 1.26.4
        scipy: 1.12.0
       Cython: 0.29.37
       pandas: 2.2.1
   matplotlib: 3.8.3
       joblib: 1.3.2
threadpoolctl: 3.3.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libopenblas
       filepath: /opt/homebrew/lib/python3.11/site-packages/numpy/.dylibs/libopenblas64_.0.dylib
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: armv8

       user_api: openmp
   internal_api: openmp
    num_threads: 16
         prefix: libomp
       filepath: /opt/homebrew/lib/python3.11/site-packages/torch/.dylibs/libomp.dylib
        version: None

       user_api: openmp
   internal_api: openmp
    num_threads: 16
         prefix: libomp
       filepath: /opt/homebrew/lib/python3.11/site-packages/sklearn/.dylibs/libomp.dylib
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 16
         prefix: libopenblas
       filepath: /opt/homebrew/lib/python3.11/site-packages/scipy/.dylibs/libopenblas.0.dylib
        version: 0.3.21.dev
threading_layer: pthreads
   architecture: armv8
```

COMMENT:
Also ran into this issue for other metrics such as "correlation". It would be good to fall back to "brute" as indicated in the docstring.

Labels: Bug
Make RFE/RFECV preserve pandas dataframes

### Describe the workflow you want to enable

Hi!

I am currently using xgboost with some categorical features. To get that to work the categorical features have to be marked as such in the pandas dataframe:
```python
df["my_cats"] = df["my_cats"].astype("string").astype("category")
```

The RFE(CV) implementation converts any input to a numpy array that can only contain numeric values which blocks me from using them with xgboost because the dtype information get's lost along the way.

So my proposal: make RFE and RFECV preserve pandas dataframes so the estimator that is used still has access to this information.


### Describe your proposed solution

I was able to get this to work with the following quick-and-dirty changes:
```python
# relax the data validation a little bit (dtype -> None, cast_to_ndarray -> False)
X, y = self._validate_data(
    X,
    y,
    accept_sparse="csc",
    ensure_min_features=2,
    force_all_finite=False,
    multi_output=True,
    dtype=None,
    cast_to_ndarray=False,
)

# make the fit command pandas compliant
estimator.fit(X.iloc[:, features], y, **fit_params)

# instead of
estimator.fit(X[:, features], y, **fit_params)
```
The second step is a hack of course and would have to be replaced depending on the input of X.

With these changes and a `set_output(transform='pandas')` i got the optimization to work.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

https://github.com/scikit-learn/scikit-learn/issues/17338 - similar problem with catboost.

COMMENT:
I'm almost sure that we got another issue about the topic. I could only find https://github.com/scikit-learn/scikit-learn/issues/25560 but there is a major difference here, we do not apply any transformation on the input data and thus only select them.

Therefore, I think that all estimators in `feature_selection` should make a copy of `X` and use `_safe_indexing` to output based on the fitted attribute `support_`.

Labels: New Feature
Make RFE/RFECV preserve pandas dataframes

### Describe the workflow you want to enable

Hi!

I am currently using xgboost with some categorical features. To get that to work the categorical features have to be marked as such in the pandas dataframe:
```python
df["my_cats"] = df["my_cats"].astype("string").astype("category")
```

The RFE(CV) implementation converts any input to a numpy array that can only contain numeric values which blocks me from using them with xgboost because the dtype information get's lost along the way.

So my proposal: make RFE and RFECV preserve pandas dataframes so the estimator that is used still has access to this information.


### Describe your proposed solution

I was able to get this to work with the following quick-and-dirty changes:
```python
# relax the data validation a little bit (dtype -> None, cast_to_ndarray -> False)
X, y = self._validate_data(
    X,
    y,
    accept_sparse="csc",
    ensure_min_features=2,
    force_all_finite=False,
    multi_output=True,
    dtype=None,
    cast_to_ndarray=False,
)

# make the fit command pandas compliant
estimator.fit(X.iloc[:, features], y, **fit_params)

# instead of
estimator.fit(X[:, features], y, **fit_params)
```
The second step is a hack of course and would have to be replaced depending on the input of X.

With these changes and a `set_output(transform='pandas')` i got the optimization to work.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

https://github.com/scikit-learn/scikit-learn/issues/17338 - similar problem with catboost.

COMMENT:
> I think that all estimators in feature_selection should make a copy of X and use _safe_indexing to output based on the fitted attribute support_.

I don't understand why you would want to make a copy if there is no inplace modification of the original input datastructure.

But I agree that we should use `_safe_indexing` to handle either dataframes or arrays with the same code while making it possible to output the same output dtype when configuring `set_output

I had a look a the source:

- the `SelectorMixin.transform` method has a `preserve_X` variable that depends on both the type of `X` and the output config: https://github.com/scikit-learn/scikit-learn/blob/69c11168dd3b178fff91aa1b16afca445bc27352/sklearn/feature_selection/_base.py#L97-L109
- the `RFE(CV).fit` method indeed always cast to arrays before passing this to the sub-estimator.

I agree we should align the behavior of the `fit` method with the logic implemented in the `SelectorMixin.transform` method.

@cleboo would you be interested in working on a PR? You can start with RFE and RFECV alone and then we could open follow-up PRs for other estimators under `sklearn.feature_selection` later if we identify similar patterns.

Labels: New Feature
Make RFE/RFECV preserve pandas dataframes

### Describe the workflow you want to enable

Hi!

I am currently using xgboost with some categorical features. To get that to work the categorical features have to be marked as such in the pandas dataframe:
```python
df["my_cats"] = df["my_cats"].astype("string").astype("category")
```

The RFE(CV) implementation converts any input to a numpy array that can only contain numeric values which blocks me from using them with xgboost because the dtype information get's lost along the way.

So my proposal: make RFE and RFECV preserve pandas dataframes so the estimator that is used still has access to this information.


### Describe your proposed solution

I was able to get this to work with the following quick-and-dirty changes:
```python
# relax the data validation a little bit (dtype -> None, cast_to_ndarray -> False)
X, y = self._validate_data(
    X,
    y,
    accept_sparse="csc",
    ensure_min_features=2,
    force_all_finite=False,
    multi_output=True,
    dtype=None,
    cast_to_ndarray=False,
)

# make the fit command pandas compliant
estimator.fit(X.iloc[:, features], y, **fit_params)

# instead of
estimator.fit(X[:, features], y, **fit_params)
```
The second step is a hack of course and would have to be replaced depending on the input of X.

With these changes and a `set_output(transform='pandas')` i got the optimization to work.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

https://github.com/scikit-learn/scikit-learn/issues/17338 - similar problem with catboost.

COMMENT:
Also, we might want to add polars support as well but we can do that in a follow-up PR to make the review process more lightweight.

Labels: New Feature
Make RFE/RFECV preserve pandas dataframes

### Describe the workflow you want to enable

Hi!

I am currently using xgboost with some categorical features. To get that to work the categorical features have to be marked as such in the pandas dataframe:
```python
df["my_cats"] = df["my_cats"].astype("string").astype("category")
```

The RFE(CV) implementation converts any input to a numpy array that can only contain numeric values which blocks me from using them with xgboost because the dtype information get's lost along the way.

So my proposal: make RFE and RFECV preserve pandas dataframes so the estimator that is used still has access to this information.


### Describe your proposed solution

I was able to get this to work with the following quick-and-dirty changes:
```python
# relax the data validation a little bit (dtype -> None, cast_to_ndarray -> False)
X, y = self._validate_data(
    X,
    y,
    accept_sparse="csc",
    ensure_min_features=2,
    force_all_finite=False,
    multi_output=True,
    dtype=None,
    cast_to_ndarray=False,
)

# make the fit command pandas compliant
estimator.fit(X.iloc[:, features], y, **fit_params)

# instead of
estimator.fit(X[:, features], y, **fit_params)
```
The second step is a hack of course and would have to be replaced depending on the input of X.

With these changes and a `set_output(transform='pandas')` i got the optimization to work.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

https://github.com/scikit-learn/scikit-learn/issues/17338 - similar problem with catboost.

COMMENT:
> I don't understand why you would want to make a copy if there is no inplace modification of the original input datastructure.

By a copy, I mean that the underlying algorithm creating the `mask` is most probably expecting working on NumPy array. We could always make sure that the underlying algorithm does not required specifically a NumPy array in which case, we then don't even need the conversion and the associated copy.

Labels: New Feature
Make RFE/RFECV preserve pandas dataframes

### Describe the workflow you want to enable

Hi!

I am currently using xgboost with some categorical features. To get that to work the categorical features have to be marked as such in the pandas dataframe:
```python
df["my_cats"] = df["my_cats"].astype("string").astype("category")
```

The RFE(CV) implementation converts any input to a numpy array that can only contain numeric values which blocks me from using them with xgboost because the dtype information get's lost along the way.

So my proposal: make RFE and RFECV preserve pandas dataframes so the estimator that is used still has access to this information.


### Describe your proposed solution

I was able to get this to work with the following quick-and-dirty changes:
```python
# relax the data validation a little bit (dtype -> None, cast_to_ndarray -> False)
X, y = self._validate_data(
    X,
    y,
    accept_sparse="csc",
    ensure_min_features=2,
    force_all_finite=False,
    multi_output=True,
    dtype=None,
    cast_to_ndarray=False,
)

# make the fit command pandas compliant
estimator.fit(X.iloc[:, features], y, **fit_params)

# instead of
estimator.fit(X[:, features], y, **fit_params)
```
The second step is a hack of course and would have to be replaced depending on the input of X.

With these changes and a `set_output(transform='pandas')` i got the optimization to work.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

https://github.com/scikit-learn/scikit-learn/issues/17338 - similar problem with catboost.

COMMENT:
Alright, thanks for the clarification. Indeed there are several occurrences of array-specific indexing such as `X[:, features]` in the RFE(CV) source code and all of them should be replaced by `_safe_indexing`.

Labels: New Feature
Make RFE/RFECV preserve pandas dataframes

### Describe the workflow you want to enable

Hi!

I am currently using xgboost with some categorical features. To get that to work the categorical features have to be marked as such in the pandas dataframe:
```python
df["my_cats"] = df["my_cats"].astype("string").astype("category")
```

The RFE(CV) implementation converts any input to a numpy array that can only contain numeric values which blocks me from using them with xgboost because the dtype information get's lost along the way.

So my proposal: make RFE and RFECV preserve pandas dataframes so the estimator that is used still has access to this information.


### Describe your proposed solution

I was able to get this to work with the following quick-and-dirty changes:
```python
# relax the data validation a little bit (dtype -> None, cast_to_ndarray -> False)
X, y = self._validate_data(
    X,
    y,
    accept_sparse="csc",
    ensure_min_features=2,
    force_all_finite=False,
    multi_output=True,
    dtype=None,
    cast_to_ndarray=False,
)

# make the fit command pandas compliant
estimator.fit(X.iloc[:, features], y, **fit_params)

# instead of
estimator.fit(X[:, features], y, **fit_params)
```
The second step is a hack of course and would have to be replaced depending on the input of X.

With these changes and a `set_output(transform='pandas')` i got the optimization to work.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

https://github.com/scikit-learn/scikit-learn/issues/17338 - similar problem with catboost.

COMMENT:
> @cleboo would you be interested in working on a PR? You can start with RFE and RFECV alone and then we could open follow-up PRs for other estimators under sklearn.feature_selection later if we identify similar patterns.

Happy to - any pointers on how to write a test for this? I guess pandas should not become an explicit dependency here?

Labels: New Feature
Make RFE/RFECV preserve pandas dataframes

### Describe the workflow you want to enable

Hi!

I am currently using xgboost with some categorical features. To get that to work the categorical features have to be marked as such in the pandas dataframe:
```python
df["my_cats"] = df["my_cats"].astype("string").astype("category")
```

The RFE(CV) implementation converts any input to a numpy array that can only contain numeric values which blocks me from using them with xgboost because the dtype information get's lost along the way.

So my proposal: make RFE and RFECV preserve pandas dataframes so the estimator that is used still has access to this information.


### Describe your proposed solution

I was able to get this to work with the following quick-and-dirty changes:
```python
# relax the data validation a little bit (dtype -> None, cast_to_ndarray -> False)
X, y = self._validate_data(
    X,
    y,
    accept_sparse="csc",
    ensure_min_features=2,
    force_all_finite=False,
    multi_output=True,
    dtype=None,
    cast_to_ndarray=False,
)

# make the fit command pandas compliant
estimator.fit(X.iloc[:, features], y, **fit_params)

# instead of
estimator.fit(X[:, features], y, **fit_params)
```
The second step is a hack of course and would have to be replaced depending on the input of X.

With these changes and a `set_output(transform='pandas')` i got the optimization to work.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

https://github.com/scikit-learn/scikit-learn/issues/17338 - similar problem with catboost.

COMMENT:
You can take inspirations from other tests that have a soft dependency on pandas, e.g. on a linux/macos machine with bash / grep:

```
git grep pandas | grep test
```

or alternatively use the "search in project files" feature of your IDE.

The most common idiom being to import pandas as follows in tests:

```python
pd = pytest.importorskip("pandas")
```

Labels: New Feature
Make RFE/RFECV preserve pandas dataframes

### Describe the workflow you want to enable

Hi!

I am currently using xgboost with some categorical features. To get that to work the categorical features have to be marked as such in the pandas dataframe:
```python
df["my_cats"] = df["my_cats"].astype("string").astype("category")
```

The RFE(CV) implementation converts any input to a numpy array that can only contain numeric values which blocks me from using them with xgboost because the dtype information get's lost along the way.

So my proposal: make RFE and RFECV preserve pandas dataframes so the estimator that is used still has access to this information.


### Describe your proposed solution

I was able to get this to work with the following quick-and-dirty changes:
```python
# relax the data validation a little bit (dtype -> None, cast_to_ndarray -> False)
X, y = self._validate_data(
    X,
    y,
    accept_sparse="csc",
    ensure_min_features=2,
    force_all_finite=False,
    multi_output=True,
    dtype=None,
    cast_to_ndarray=False,
)

# make the fit command pandas compliant
estimator.fit(X.iloc[:, features], y, **fit_params)

# instead of
estimator.fit(X[:, features], y, **fit_params)
```
The second step is a hack of course and would have to be replaced depending on the input of X.

With these changes and a `set_output(transform='pandas')` i got the optimization to work.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

https://github.com/scikit-learn/scikit-learn/issues/17338 - similar problem with catboost.

COMMENT:
Hi @cleboo, Have you made any progress on this? I would love to see this problem solved as well. If you are working on a solution, let me know, and I would try to help you.

Labels: New Feature
Make RFE/RFECV preserve pandas dataframes

### Describe the workflow you want to enable

Hi!

I am currently using xgboost with some categorical features. To get that to work the categorical features have to be marked as such in the pandas dataframe:
```python
df["my_cats"] = df["my_cats"].astype("string").astype("category")
```

The RFE(CV) implementation converts any input to a numpy array that can only contain numeric values which blocks me from using them with xgboost because the dtype information get's lost along the way.

So my proposal: make RFE and RFECV preserve pandas dataframes so the estimator that is used still has access to this information.


### Describe your proposed solution

I was able to get this to work with the following quick-and-dirty changes:
```python
# relax the data validation a little bit (dtype -> None, cast_to_ndarray -> False)
X, y = self._validate_data(
    X,
    y,
    accept_sparse="csc",
    ensure_min_features=2,
    force_all_finite=False,
    multi_output=True,
    dtype=None,
    cast_to_ndarray=False,
)

# make the fit command pandas compliant
estimator.fit(X.iloc[:, features], y, **fit_params)

# instead of
estimator.fit(X[:, features], y, **fit_params)
```
The second step is a hack of course and would have to be replaced depending on the input of X.

With these changes and a `set_output(transform='pandas')` i got the optimization to work.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

https://github.com/scikit-learn/scikit-learn/issues/17338 - similar problem with catboost.

COMMENT:
A temporary workaround which I imagine will solve the OP's issue is to make a pipeline of the RFE and then a new classifier. The RFE will be responsible for transforming the X to Xt, which it does preserving dataframe status. The new classifier will then use Xt for fitting and predicting, in both cases receiving a dataframe. I.e. instead of using the classifier which RFE returns, which loses feature names (and categorical cols), we just use RFE for the transform and then fit a new classifer to handle the predict.  

So
```python
import sklearn.pipeline

estimator = sklearn.tree.DecisionTreeClassifier()
# estimator = sklearn.linear_model.LogisticRegression()

rfe = sklearn.feature_selection.RFE(
        estimator=estimator,
        n_features_to_select=2
        ).set_output(transform='pandas')

clf = sklearn.pipeline.make_pipeline(rfe, estimator)
clf.fit(X_train,y_train)
print(clf[0].transform(X_test)[:1])
print(clf.predict(X_test))
print(clf.feature_names_in_)
print(clf[0].get_feature_names_out())
print(clf[1].feature_names_in_)
```
```
    petal length (cm)  petal width (cm)
85                4.5               1.6
[1 1 0 0 1 1 2 0 0 2 2 2 2 1 1 1 2 1 1 0 2 1 1 1 0 1 2 2 1 2]
['sepal length (cm)' 'sepal width (cm)' 'petal length (cm)' 'petal width (cm)']
['petal length (cm)' 'petal width (cm)']
['petal length (cm)' 'petal width (cm)']
```
- the classifier has the feature names as desired
- this would presumably also work for OP's issue by preserving categorical columns.

Labels: New Feature
Add an option handle_unknown="warn" in OneHotEncoder

Follow-up to https://github.com/scikit-learn/scikit-learn/pull/16881

It seems that it could be interested to log an eventual detection of new category during inference and issue a warning instead of silently ignoring them.

Therefore, it seems reasonable to add a new option `handle_unknown="warn"` to `OneHotEncoder` that should behave as `"ignore"` but should issue an additional warning.

COMMENT:
Seems handy! I can work on it, if it will be approved.

Some questions:
Should it log just the detection of the new value in any feature (like a boolean flag) or should it warn every time it sees the new value in every feature?
Should it just log or maybe also propose a solution to mitigate an unwanted behaviours?

Labels: Enhancement