Talip7/scikit-learn-issues-multilabel huggingface dataset:
False positive warning in `FunctionTransformer`

While looking at #26543, I find out that we raise a `FutureWarning` that looked like a false positive to me:

```python
from sklearn.preprocessing import FunctionTransformer, OneHotEncoder
from sklearn.pipeline import make_pipeline
import pandas as pd
import numpy as np


test_df = pd.DataFrame(
    np.random.randint(low=0, high=9, size=(10, 2)), columns=["None", "valid"]
)


def _drop_None_cols(df):
    col_names = [col for col in df.columns if "None" in col]
    if len(col_names):
        return df.drop(columns=col_names)
    return df


encoder_dropping_None = make_pipeline(
    OneHotEncoder(sparse_output=False),
    FunctionTransformer(_drop_None_cols),
).set_output(transform="pandas")


# encoder_dropping_None.fit_transform(test_df)
```

```
/Users/glemaitre/Documents/packages/scikit-learn/sklearn/preprocessing/_function_transformer.py:345: UserWarning: With transform="pandas", `func` should return a DataFrame to follow the set_output API.
  warnings.warn(
```

Indeed, my function `func` is returning a dataframe and the output of the transform is indeed a dataframe. So the warning is a bit surprising. Using the global config (i.e. `set_config` will not raise this warning).

COMMENT:
We might want to delay the raise of the warning `FunctionTransformer` at the end of its `.transform` if the concrete type of its return value is not consistent with the currently configured `.set_output` policy instead of at the time where we call `.set_output` on it.

Labels: Enhancement
False positive warning in `FunctionTransformer`

While looking at #26543, I find out that we raise a `FutureWarning` that looked like a false positive to me:

```python
from sklearn.preprocessing import FunctionTransformer, OneHotEncoder
from sklearn.pipeline import make_pipeline
import pandas as pd
import numpy as np


test_df = pd.DataFrame(
    np.random.randint(low=0, high=9, size=(10, 2)), columns=["None", "valid"]
)


def _drop_None_cols(df):
    col_names = [col for col in df.columns if "None" in col]
    if len(col_names):
        return df.drop(columns=col_names)
    return df


encoder_dropping_None = make_pipeline(
    OneHotEncoder(sparse_output=False),
    FunctionTransformer(_drop_None_cols),
).set_output(transform="pandas")


# encoder_dropping_None.fit_transform(test_df)
```

```
/Users/glemaitre/Documents/packages/scikit-learn/sklearn/preprocessing/_function_transformer.py:345: UserWarning: With transform="pandas", `func` should return a DataFrame to follow the set_output API.
  warnings.warn(
```

Indeed, my function `func` is returning a dataframe and the output of the transform is indeed a dataframe. So the warning is a bit surprising. Using the global config (i.e. `set_config` will not raise this warning).

COMMENT:
It seems that `FunctionTransformer` do not have `get_feature_names_out` if `feature_names_out` is provided, so that `set_output` is not called and `_sklearn_output_config` is not set. It is then not feasible to use `_get_output_config` to get the output policy unless setting global config. Am I not understanding you correctly @ogrisel?

Labels: Enhancement
False positive warning in `FunctionTransformer`

While looking at #26543, I find out that we raise a `FutureWarning` that looked like a false positive to me:

```python
from sklearn.preprocessing import FunctionTransformer, OneHotEncoder
from sklearn.pipeline import make_pipeline
import pandas as pd
import numpy as np


test_df = pd.DataFrame(
    np.random.randint(low=0, high=9, size=(10, 2)), columns=["None", "valid"]
)


def _drop_None_cols(df):
    col_names = [col for col in df.columns if "None" in col]
    if len(col_names):
        return df.drop(columns=col_names)
    return df


encoder_dropping_None = make_pipeline(
    OneHotEncoder(sparse_output=False),
    FunctionTransformer(_drop_None_cols),
).set_output(transform="pandas")


# encoder_dropping_None.fit_transform(test_df)
```

```
/Users/glemaitre/Documents/packages/scikit-learn/sklearn/preprocessing/_function_transformer.py:345: UserWarning: With transform="pandas", `func` should return a DataFrame to follow the set_output API.
  warnings.warn(
```

Indeed, my function `func` is returning a dataframe and the output of the transform is indeed a dataframe. So the warning is a bit surprising. Using the global config (i.e. `set_config` will not raise this warning).

COMMENT:
I agree the warning is a false positive. I opened https://github.com/scikit-learn/scikit-learn/pull/26944 to fix it.

Labels: Enhancement
AttributeError: module 'sklearn.metrics._dist_metrics' has no attribute 'DistanceMetric32'

### Describe the bug

I am using the pyldavis library which depends on scikit learn, I have been working for months without any problem. Today at night a new error appeared when I tried the library

`import pyLDAvis`

AttributeError: module 'sklearn.metrics._dist_metrics' has no attribute 'DistanceMetric32'.

If I try to import the function directly 

`from sklearn.metrics import _dist_metrics`


 it returns the same error

### Steps/Code to Reproduce

`from sklearn.metrics import _dist_metrics`

### Expected Results

No error is thrown

### Actual Results

File sklearn/metrics/_pairwise_distances_reduction/_base.pyx:1, in init sklearn.metrics._pairwise_distances_reduction._base()

AttributeError: module 'sklearn.metrics._dist_metrics' has no attribute 'DistanceMetric32'

### Versions

```shell
System:
    python: 3.10.6 | packaged by conda-forge | (main, Aug 22 2022, 20:35:26) [GCC 10.4.0]
executable: /home/trusted-service-user/cluster-env/env/bin/python
   machine: Linux-4.15.0-1165-azure-x86_64-with-glibc2.27

Python dependencies:
      sklearn: 1.1.3
          pip: 22.3.1
   setuptools: 65.5.1
        numpy: 1.23.4
        scipy: 1.9.3
       Cython: 0.29.32
       pandas: 1.5.1
   matplotlib: 3.6.2
       joblib: 1.2.0
threadpoolctl: 3.1.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: mkl
         prefix: libmkl_rt
       filepath: /home/trusted-service-user/cluster-env/env/lib/libmkl_rt.so.2
        version: 2022.1-Product
threading_layer: intel
    num_threads: 8

       user_api: openmp
   internal_api: openmp
         prefix: libomp
       filepath: /home/trusted-service-user/cluster-env/env/lib/libomp.so
        version: None
    num_threads: 16

       user_api: openmp
   internal_api: openmp
         prefix: libgomp
       filepath: /home/trusted-service-user/cluster-env/env/lib/python3.10/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
    num_threads: 16
```

COMMENT:
Also note that the failing import is coming from a private import. If pyLDAvis is somehow importing private methods, they need to fix the issue on their side. Closing for now, happy to re-open if a bug on our side is confirmed.

Labels: Bug, Needs Triage
What happend to the idea of adding a 'handle_missing' parameter to the OneHotEncoder?

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26531

<div type='discussions-op-text'>

<sup>Originally posted by **woodly0** June  7, 2023</sup>
Hello,
I'm having trouble understanding what finally happened to the idea of introducing a `handle_missing` parameter for the `OneHotEncoder`. My current project could still benefit from such an implementation.
There are many existing issues regarding this topic, however, I cannot deduct what was finally decided/implemented and what wasn't.
- #11996
- #12025
- #17317
- #23436

Considering the following features:
```py
import pandas as pd

test_df = pd.DataFrame(
    {"col1": ["red", "blue", "blue"], "col2": ["car", None, "plane"]}
)
```
when using the encoder:
```py
from sklearn.preprocessing import OneHotEncoder

ohe = OneHotEncoder(
    handle_unknown="ignore",
    sparse_output=False,
    #handle_missing="ignore"
)
ohe.fit_transform(test_df)
```
I get the output:
```
array([[0., 1., 1., 0., 0.],
       [1., 0., 0., 0., 1.],
       [1., 0., 0., 1., 0.]])
```
but what I'm actually looking for is to remove the `None`, i.e. not create a new feature but set all the others to zero:
```
array([[0., 1., 1., 0.],
       [1., 0., 0., 0.],
       [1., 0., 0., 1.]])
```
Is there a way to achieve this without using another transformer object?</div>

COMMENT:
I am not really keen to drop missing values with an option. I would prefer some code allowing to make it more explicit than just an option.

We currently can do that:

```python
import sklearn
from sklearn.preprocessing import FunctionTransformer, OneHotEncoder
from sklearn.pipeline import make_pipeline

sklearn.set_config(transform_output="pandas")


def _drop_None_cols(df):
    col_names = [col for col in df.columns if "None" in col]
    if len(col_names):
        return df.drop(columns=col_names)
    return df


encoder_dropping_None = make_pipeline(
    OneHotEncoder(sparse_output=False),
    FunctionTransformer(_drop_None_cols),
)
encoder_dropping_None.fit_transform(test_df)
```

  | col1_blue | col1_red | col2_car | col2_plane
-- | -- | -- | -- | --
0|0.0 | 1.0 | 1.0 | 0.0
1|1.0 | 0.0 | 0.0 | 0.0
2|1.0 | 0.0 | 0.0 | 1.0

Labels: Enhancement
What happend to the idea of adding a 'handle_missing' parameter to the OneHotEncoder?

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26531

<div type='discussions-op-text'>

<sup>Originally posted by **woodly0** June  7, 2023</sup>
Hello,
I'm having trouble understanding what finally happened to the idea of introducing a `handle_missing` parameter for the `OneHotEncoder`. My current project could still benefit from such an implementation.
There are many existing issues regarding this topic, however, I cannot deduct what was finally decided/implemented and what wasn't.
- #11996
- #12025
- #17317
- #23436

Considering the following features:
```py
import pandas as pd

test_df = pd.DataFrame(
    {"col1": ["red", "blue", "blue"], "col2": ["car", None, "plane"]}
)
```
when using the encoder:
```py
from sklearn.preprocessing import OneHotEncoder

ohe = OneHotEncoder(
    handle_unknown="ignore",
    sparse_output=False,
    #handle_missing="ignore"
)
ohe.fit_transform(test_df)
```
I get the output:
```
array([[0., 1., 1., 0., 0.],
       [1., 0., 0., 0., 1.],
       [1., 0., 0., 1., 0.]])
```
but what I'm actually looking for is to remove the `None`, i.e. not create a new feature but set all the others to zero:
```
array([[0., 1., 1., 0.],
       [1., 0., 0., 0.],
       [1., 0., 0., 1.]])
```
Is there a way to achieve this without using another transformer object?</div>

COMMENT:
I think it would make sense to have:

- `handle_missing="encode_as_category"` by default with an option to set it to `handle_ignore="ignore"`,
- `missing_marker=(np.nan, pd.NA, None)` which make it possible to specify which values should be consider missing value markers.

The `missing_marker` parameter would only be used if `handle_ignore="ignore"` to avoid creating output columns for such values.

Labels: Enhancement
What happend to the idea of adding a 'handle_missing' parameter to the OneHotEncoder?

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26531

<div type='discussions-op-text'>

<sup>Originally posted by **woodly0** June  7, 2023</sup>
Hello,
I'm having trouble understanding what finally happened to the idea of introducing a `handle_missing` parameter for the `OneHotEncoder`. My current project could still benefit from such an implementation.
There are many existing issues regarding this topic, however, I cannot deduct what was finally decided/implemented and what wasn't.
- #11996
- #12025
- #17317
- #23436

Considering the following features:
```py
import pandas as pd

test_df = pd.DataFrame(
    {"col1": ["red", "blue", "blue"], "col2": ["car", None, "plane"]}
)
```
when using the encoder:
```py
from sklearn.preprocessing import OneHotEncoder

ohe = OneHotEncoder(
    handle_unknown="ignore",
    sparse_output=False,
    #handle_missing="ignore"
)
ohe.fit_transform(test_df)
```
I get the output:
```
array([[0., 1., 1., 0., 0.],
       [1., 0., 0., 0., 1.],
       [1., 0., 0., 1., 0.]])
```
but what I'm actually looking for is to remove the `None`, i.e. not create a new feature but set all the others to zero:
```
array([[0., 1., 1., 0.],
       [1., 0., 0., 0.],
       [1., 0., 0., 1.]])
```
Is there a way to achieve this without using another transformer object?</div>

COMMENT:
@glemaitre: Thanks for your answer. What you suggest is what I am actually doing already. 
My question is about including this in the `OneHotEncoder` itself and thus avoiding the additional `FunctionTransformer`. 

In order to avoid collinearity, I could use the parameter `drop='first'` but it seems more intuitive to drop the "_None" feature instead. Don't you agree? It is what the `pandas.get_dummies()` function does by default.

Labels: Enhancement
What happend to the idea of adding a 'handle_missing' parameter to the OneHotEncoder?

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26531

<div type='discussions-op-text'>

<sup>Originally posted by **woodly0** June  7, 2023</sup>
Hello,
I'm having trouble understanding what finally happened to the idea of introducing a `handle_missing` parameter for the `OneHotEncoder`. My current project could still benefit from such an implementation.
There are many existing issues regarding this topic, however, I cannot deduct what was finally decided/implemented and what wasn't.
- #11996
- #12025
- #17317
- #23436

Considering the following features:
```py
import pandas as pd

test_df = pd.DataFrame(
    {"col1": ["red", "blue", "blue"], "col2": ["car", None, "plane"]}
)
```
when using the encoder:
```py
from sklearn.preprocessing import OneHotEncoder

ohe = OneHotEncoder(
    handle_unknown="ignore",
    sparse_output=False,
    #handle_missing="ignore"
)
ohe.fit_transform(test_df)
```
I get the output:
```
array([[0., 1., 1., 0., 0.],
       [1., 0., 0., 0., 1.],
       [1., 0., 0., 1., 0.]])
```
but what I'm actually looking for is to remove the `None`, i.e. not create a new feature but set all the others to zero:
```
array([[0., 1., 1., 0.],
       [1., 0., 0., 0.],
       [1., 0., 0., 1.]])
```
Is there a way to achieve this without using another transformer object?</div>

COMMENT:
@ogrisel: Sorry, I'm not sure to understand what you are suggesting.

Labels: Enhancement
What happend to the idea of adding a 'handle_missing' parameter to the OneHotEncoder?

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26531

<div type='discussions-op-text'>

<sup>Originally posted by **woodly0** June  7, 2023</sup>
Hello,
I'm having trouble understanding what finally happened to the idea of introducing a `handle_missing` parameter for the `OneHotEncoder`. My current project could still benefit from such an implementation.
There are many existing issues regarding this topic, however, I cannot deduct what was finally decided/implemented and what wasn't.
- #11996
- #12025
- #17317
- #23436

Considering the following features:
```py
import pandas as pd

test_df = pd.DataFrame(
    {"col1": ["red", "blue", "blue"], "col2": ["car", None, "plane"]}
)
```
when using the encoder:
```py
from sklearn.preprocessing import OneHotEncoder

ohe = OneHotEncoder(
    handle_unknown="ignore",
    sparse_output=False,
    #handle_missing="ignore"
)
ohe.fit_transform(test_df)
```
I get the output:
```
array([[0., 1., 1., 0., 0.],
       [1., 0., 0., 0., 1.],
       [1., 0., 0., 1., 0.]])
```
but what I'm actually looking for is to remove the `None`, i.e. not create a new feature but set all the others to zero:
```
array([[0., 1., 1., 0.],
       [1., 0., 0., 0.],
       [1., 0., 0., 1.]])
```
Is there a way to achieve this without using another transformer object?</div>

COMMENT:
@woodly0  @ogrisel is suggesting that there should be a parameter to handle (null or None or pd. NA) values or we can give values that can be treated like None.

Labels: Enhancement
What happend to the idea of adding a 'handle_missing' parameter to the OneHotEncoder?

### Discussed in https://github.com/scikit-learn/scikit-learn/discussions/26531

<div type='discussions-op-text'>

<sup>Originally posted by **woodly0** June  7, 2023</sup>
Hello,
I'm having trouble understanding what finally happened to the idea of introducing a `handle_missing` parameter for the `OneHotEncoder`. My current project could still benefit from such an implementation.
There are many existing issues regarding this topic, however, I cannot deduct what was finally decided/implemented and what wasn't.
- #11996
- #12025
- #17317
- #23436

Considering the following features:
```py
import pandas as pd

test_df = pd.DataFrame(
    {"col1": ["red", "blue", "blue"], "col2": ["car", None, "plane"]}
)
```
when using the encoder:
```py
from sklearn.preprocessing import OneHotEncoder

ohe = OneHotEncoder(
    handle_unknown="ignore",
    sparse_output=False,
    #handle_missing="ignore"
)
ohe.fit_transform(test_df)
```
I get the output:
```
array([[0., 1., 1., 0., 0.],
       [1., 0., 0., 0., 1.],
       [1., 0., 0., 1., 0.]])
```
but what I'm actually looking for is to remove the `None`, i.e. not create a new feature but set all the others to zero:
```
array([[0., 1., 1., 0.],
       [1., 0., 0., 0.],
       [1., 0., 0., 1.]])
```
Is there a way to achieve this without using another transformer object?</div>

COMMENT:
I am willing to work on that and based on @ogrisel comment I'm thinking of adding two parameters:

    handle_missing : {'encode_as_category', 'ignore'}, default='encode_as_category'
        Specifies the strategy to handle missing values during encoding.

        - If "encode_as_category", missing values are treated as a separate category and encoded into distinct
          columns, regardless of the `missing_values` parameter.
        - If "ignore", missing values are excluded from the encoding process, and no column is created for them
          in the output.

    missing_values : int, float, str, np.nan or None, default=np.nan
        The placeholder to identify missing values when `handle_missing="ignore"`. All values equal to
        `missing_values` will be considered missing and ignored during encoding.

        This parameter has no effect when `handle_missing="encode_as_category"`.


I chose the name missing_values for consistency with sklearn.impute, and the description aligns with their style.

I'll be waiting for feedback on that and will start working on it if it looks good to everyone.

Labels: Enhancement
HDBSCAN tests are failing on pypy

As seen in the nigthly build, a lot of hdbscan tests are failing in the pypy job.
Here's a link with the full failure report for some of them https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=55668&view=logs&jobId=0b16f832-29d6-5b92-1c23-eb006f606a66&j=0b16f832-29d6-5b92-1c23-eb006f606a66&t=0cdfadab-878f-502c-9df1-54a896e98c4f
Looks like the cause is always the same, but I couldn't figure out what it really is and how to solve it.

cc/ @Micky774

COMMENT:
I'm starting to take a look at these. This should be fun 😅

Labels: pypy
ValueError: The covariance matrix of the support data is equal to 0 - Elliptic Envelope

### Describe the bug

I have been using Elliptic Envelope on a simple time series dataset with default parameters and only setting `contamination` value to `0.5`. However it throws me error like below:

`ValueError: The covariance matrix of the support data is equal to 0, try to increase support_fraction`

Looking at the error, I referred to the documentation and the equation of `support_fraction` does not seem to be clear. It sayd ->  `[n_sample + n_features + 1] / 2`. `Range is (0, 1)`. The value of `n_sample` and `n_features` can never be such that it will fall within 0 and 1. Could you please help in understanding and how this issue can be resolved?

### Steps/Code to Reproduce

```py
model = EllipticEnvelope(contamination=0.5)
model.fit(X)
```

### Expected Results

No error is thrown

### Actual Results

```
File /opt/anaconda3/envs/benchmark/lib/python3.9/site-packages/sklearn/covariance/_elliptic_envelope.py:145, in EllipticEnvelope.fit(self, X, y)
    134 def fit(self, X, y=None):
    135     """Fit the EllipticEnvelope model.
    136 
    137     Parameters
   (...)
    143         Not used, present for API consistency by convention.
    144     """
--> 145     super().fit(X)
    146     self.offset_ = np.percentile(-self.dist_, 100. * self.contamination)
    147     return self

File /opt/anaconda3/envs/benchmark/lib/python3.9/site-packages/sklearn/covariance/_robust_covariance.py:668, in MinCovDet.fit(self, X, y)
    666 self.dist_ = raw_dist
    667 # obtain consistency at normal models
--> 668 self.correct_covariance(X)
    669 # re-weight estimator
    670 self.reweight_covariance(X)

File /opt/anaconda3/envs/benchmark/lib/python3.9/site-packages/sklearn/covariance/_robust_covariance.py:705, in MinCovDet.correct_covariance(self, data)
    703 n_support = np.sum(self.support_)
    704 if n_support < n_samples and np.allclose(self.raw_covariance_, 0):
--> 705     raise ValueError('The covariance matrix of the support data '
    706                      'is equal to 0, try to increase support_fraction')
    707 correction = np.median(self.dist_) / chi2(data.shape[1]).isf(0.5)
    708 covariance_corrected = self.raw_covariance_ * correction

ValueError: The covariance matrix of the support data is equal to 0, try to increase support_fraction
```

### Versions

```shell
0.24.1
```

COMMENT:
Thank you for reporting this. Could you please extend your "Steps/Code to Reproduce" with a (minimal) `X` that reproduces the problem?

Labels: Bug, Needs Info