Talip7/scikit-learn-issues-multilabel huggingface dataset:
make_classification: repeated features are not chosen uniformly at random

I have just noticed that this piece of code in `make_classification` which selects features to repeat does not do so uniformly at random:
https://github.com/scikit-learn/scikit-learn/blob/9aaed498795f68e5956ea762fef9c440ca9eb239/sklearn/datasets/_samples_generator.py#L255-L259
It selects the first and last feature with half the probability of the other features, as `(n-1)*...+0.5` ranges from 0.5 to n-0.5, so 0 and n-1 only have half the probability of being selected compared to 1, 2, ..., n-2.  A corrected version would read:
```python
        indices = (n * generator.uniform(size=n_repeated)).astype(np.intp)
```
Making this correction, though, would cause `make_classification` to probably produce different output if `n_repeated > 0`, so I do not know how to fix at this point; I haven't checked whether `scikit-learn` makes any guarantees about reproducibility between versions.

COMMENT:
Oh yes, that's much cleaner!

I can do a PR (though it would not be for a few days).  It would be good to have another look at #25908 again; if it is agreed that some changes are needed here for the reasons I indicate, then making two output-changing changes at the same time would be more friendly than changing them on two separate occasions.

Labels: Needs Triage
make_classification: repeated features are not chosen uniformly at random

I have just noticed that this piece of code in `make_classification` which selects features to repeat does not do so uniformly at random:
https://github.com/scikit-learn/scikit-learn/blob/9aaed498795f68e5956ea762fef9c440ca9eb239/sklearn/datasets/_samples_generator.py#L255-L259
It selects the first and last feature with half the probability of the other features, as `(n-1)*...+0.5` ranges from 0.5 to n-0.5, so 0 and n-1 only have half the probability of being selected compared to 1, 2, ..., n-2.  A corrected version would read:
```python
        indices = (n * generator.uniform(size=n_repeated)).astype(np.intp)
```
Making this correction, though, would cause `make_classification` to probably produce different output if `n_repeated > 0`, so I do not know how to fix at this point; I haven't checked whether `scikit-learn` makes any guarantees about reproducibility between versions.

COMMENT:
Actually, reflecting on what you said in #25908, since this is not intended to be used as a robust simulation, but only as a "some kind of dataset, quick", I really don't think it matters that it's not picking uniformly.  So I'm happy to leave the code as-is.

Labels: Needs Triage
make_classification: repeated features are not chosen uniformly at random

I have just noticed that this piece of code in `make_classification` which selects features to repeat does not do so uniformly at random:
https://github.com/scikit-learn/scikit-learn/blob/9aaed498795f68e5956ea762fef9c440ca9eb239/sklearn/datasets/_samples_generator.py#L255-L259
It selects the first and last feature with half the probability of the other features, as `(n-1)*...+0.5` ranges from 0.5 to n-0.5, so 0 and n-1 only have half the probability of being selected compared to 1, 2, ..., n-2.  A corrected version would read:
```python
        indices = (n * generator.uniform(size=n_repeated)).astype(np.intp)
```
Making this correction, though, would cause `make_classification` to probably produce different output if `n_repeated > 0`, so I do not know how to fix at this point; I haven't checked whether `scikit-learn` makes any guarantees about reproducibility between versions.

COMMENT:
That sounds reasonable. Despite my label changing action just now, I think closing the issue is a good way forward unless someone has a strong opinion.

While the fix might be simple (hence "good first issue") we might have to add a deprecation cycle, etc which then starts to be a lot of code and effort for not much(?) gain.

WDYT?

Labels: Needs Triage
make_classification: repeated features are not chosen uniformly at random

I have just noticed that this piece of code in `make_classification` which selects features to repeat does not do so uniformly at random:
https://github.com/scikit-learn/scikit-learn/blob/9aaed498795f68e5956ea762fef9c440ca9eb239/sklearn/datasets/_samples_generator.py#L255-L259
It selects the first and last feature with half the probability of the other features, as `(n-1)*...+0.5` ranges from 0.5 to n-0.5, so 0 and n-1 only have half the probability of being selected compared to 1, 2, ..., n-2.  A corrected version would read:
```python
        indices = (n * generator.uniform(size=n_repeated)).astype(np.intp)
```
Making this correction, though, would cause `make_classification` to probably produce different output if `n_repeated > 0`, so I do not know how to fix at this point; I haven't checked whether `scikit-learn` makes any guarantees about reproducibility between versions.

COMMENT:
Ok, since we haven't really documented what distribution we use, we could overlook this one. And indeed the fix would make sense, but it might break some code, and might not be worth it. So closing this. Thanks for the report and the investigation though.

Labels: Needs Triage
Support other dataframes like polars and pyarrow not just pandas

### Describe the workflow you want to enable

Currently, scikit-learn nowhere claims to support [pyarrow](https://arrow.apache.org/docs/python/) or [polars](https://www.pola.rs/). And indeed,
```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler, KBinsDiscretizer
from sklearn.compose import ColumnTransformer

X, y = load_iris(as_frame=True, return_X_y=True)
sepal_cols = ["sepal length (cm)", "sepal width (cm)"]
petal_cols = ["petal length (cm)", "petal width (cm)"]

preprocessor = ColumnTransformer(
    [
        ("scaler", StandardScaler(), sepal_cols),
        ("kbin", KBinsDiscretizer(encode="ordinal"), petal_cols),
    ],
    verbose_feature_names_out=False,
)

import polars as pl  # or import pyarrow as pa
X_pl = pl.from_pandas(X)  # or X_pa = pa.table(X)

preprocessor.fit_transform(X_pl)
# preprocessor.set_output(transform="pandas").fit_transform(X_pl)
```

errors with
```
AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

ValueError: Specifying the columns using strings is only supported for pandas DataFrames
```

### Describe your proposed solution

scikit-learn should support those dataframes, maybe via the [python dataframe interchange protocol](https://data-apis.org/dataframe-protocol/latest/index.html).

In that regard, a new option like `set_output(transform="dataframe")` would be nice.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Some related discussion came up in #25813.

COMMENT:
We definitely should fix this, I'm not sure if @thomasjpfan already has plans for it.

Labels: New Feature, RFC
Support other dataframes like polars and pyarrow not just pandas

### Describe the workflow you want to enable

Currently, scikit-learn nowhere claims to support [pyarrow](https://arrow.apache.org/docs/python/) or [polars](https://www.pola.rs/). And indeed,
```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler, KBinsDiscretizer
from sklearn.compose import ColumnTransformer

X, y = load_iris(as_frame=True, return_X_y=True)
sepal_cols = ["sepal length (cm)", "sepal width (cm)"]
petal_cols = ["petal length (cm)", "petal width (cm)"]

preprocessor = ColumnTransformer(
    [
        ("scaler", StandardScaler(), sepal_cols),
        ("kbin", KBinsDiscretizer(encode="ordinal"), petal_cols),
    ],
    verbose_feature_names_out=False,
)

import polars as pl  # or import pyarrow as pa
X_pl = pl.from_pandas(X)  # or X_pa = pa.table(X)

preprocessor.fit_transform(X_pl)
# preprocessor.set_output(transform="pandas").fit_transform(X_pl)
```

errors with
```
AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

ValueError: Specifying the columns using strings is only supported for pandas DataFrames
```

### Describe your proposed solution

scikit-learn should support those dataframes, maybe via the [python dataframe interchange protocol](https://data-apis.org/dataframe-protocol/latest/index.html).

In that regard, a new option like `set_output(transform="dataframe")` would be nice.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Some related discussion came up in #25813.

COMMENT:
I think it would make a lot of sense to support other popular data frames, especially if they support the data frame protocol.

> I'm not sure if @thomasjpfan already has plans for it.

If people have plans to work on things like this, it would be great to share them before they start working on it. Seems like a good opportunity to get collaboration going.

Labels: New Feature, RFC
Support other dataframes like polars and pyarrow not just pandas

### Describe the workflow you want to enable

Currently, scikit-learn nowhere claims to support [pyarrow](https://arrow.apache.org/docs/python/) or [polars](https://www.pola.rs/). And indeed,
```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler, KBinsDiscretizer
from sklearn.compose import ColumnTransformer

X, y = load_iris(as_frame=True, return_X_y=True)
sepal_cols = ["sepal length (cm)", "sepal width (cm)"]
petal_cols = ["petal length (cm)", "petal width (cm)"]

preprocessor = ColumnTransformer(
    [
        ("scaler", StandardScaler(), sepal_cols),
        ("kbin", KBinsDiscretizer(encode="ordinal"), petal_cols),
    ],
    verbose_feature_names_out=False,
)

import polars as pl  # or import pyarrow as pa
X_pl = pl.from_pandas(X)  # or X_pa = pa.table(X)

preprocessor.fit_transform(X_pl)
# preprocessor.set_output(transform="pandas").fit_transform(X_pl)
```

errors with
```
AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

ValueError: Specifying the columns using strings is only supported for pandas DataFrames
```

### Describe your proposed solution

scikit-learn should support those dataframes, maybe via the [python dataframe interchange protocol](https://data-apis.org/dataframe-protocol/latest/index.html).

In that regard, a new option like `set_output(transform="dataframe")` would be nice.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Some related discussion came up in #25813.

COMMENT:
@thomasjpfan Hello Thomas, has this matter been left for further discussion? Am I permitted to take it?

Labels: New Feature, RFC
Support other dataframes like polars and pyarrow not just pandas

### Describe the workflow you want to enable

Currently, scikit-learn nowhere claims to support [pyarrow](https://arrow.apache.org/docs/python/) or [polars](https://www.pola.rs/). And indeed,
```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler, KBinsDiscretizer
from sklearn.compose import ColumnTransformer

X, y = load_iris(as_frame=True, return_X_y=True)
sepal_cols = ["sepal length (cm)", "sepal width (cm)"]
petal_cols = ["petal length (cm)", "petal width (cm)"]

preprocessor = ColumnTransformer(
    [
        ("scaler", StandardScaler(), sepal_cols),
        ("kbin", KBinsDiscretizer(encode="ordinal"), petal_cols),
    ],
    verbose_feature_names_out=False,
)

import polars as pl  # or import pyarrow as pa
X_pl = pl.from_pandas(X)  # or X_pa = pa.table(X)

preprocessor.fit_transform(X_pl)
# preprocessor.set_output(transform="pandas").fit_transform(X_pl)
```

errors with
```
AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

ValueError: Specifying the columns using strings is only supported for pandas DataFrames
```

### Describe your proposed solution

scikit-learn should support those dataframes, maybe via the [python dataframe interchange protocol](https://data-apis.org/dataframe-protocol/latest/index.html).

In that regard, a new option like `set_output(transform="dataframe")` would be nice.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Some related discussion came up in #25813.

COMMENT:
@jiawei-zhang-a this is by far not a good first issue, and we need to discuss further. I suggest other simpler issues to start with. But happy that you're looking to contribute here :)

Labels: New Feature, RFC
Support other dataframes like polars and pyarrow not just pandas

### Describe the workflow you want to enable

Currently, scikit-learn nowhere claims to support [pyarrow](https://arrow.apache.org/docs/python/) or [polars](https://www.pola.rs/). And indeed,
```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler, KBinsDiscretizer
from sklearn.compose import ColumnTransformer

X, y = load_iris(as_frame=True, return_X_y=True)
sepal_cols = ["sepal length (cm)", "sepal width (cm)"]
petal_cols = ["petal length (cm)", "petal width (cm)"]

preprocessor = ColumnTransformer(
    [
        ("scaler", StandardScaler(), sepal_cols),
        ("kbin", KBinsDiscretizer(encode="ordinal"), petal_cols),
    ],
    verbose_feature_names_out=False,
)

import polars as pl  # or import pyarrow as pa
X_pl = pl.from_pandas(X)  # or X_pa = pa.table(X)

preprocessor.fit_transform(X_pl)
# preprocessor.set_output(transform="pandas").fit_transform(X_pl)
```

errors with
```
AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

ValueError: Specifying the columns using strings is only supported for pandas DataFrames
```

### Describe your proposed solution

scikit-learn should support those dataframes, maybe via the [python dataframe interchange protocol](https://data-apis.org/dataframe-protocol/latest/index.html).

In that regard, a new option like `set_output(transform="dataframe")` would be nice.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Some related discussion came up in #25813.

COMMENT:
@adrinjalali Your words are greatly appreciated, and I am excited at the opportunity to contribute to the project. Thank you for your encouragement!

Labels: New Feature, RFC
Support other dataframes like polars and pyarrow not just pandas

### Describe the workflow you want to enable

Currently, scikit-learn nowhere claims to support [pyarrow](https://arrow.apache.org/docs/python/) or [polars](https://www.pola.rs/). And indeed,
```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler, KBinsDiscretizer
from sklearn.compose import ColumnTransformer

X, y = load_iris(as_frame=True, return_X_y=True)
sepal_cols = ["sepal length (cm)", "sepal width (cm)"]
petal_cols = ["petal length (cm)", "petal width (cm)"]

preprocessor = ColumnTransformer(
    [
        ("scaler", StandardScaler(), sepal_cols),
        ("kbin", KBinsDiscretizer(encode="ordinal"), petal_cols),
    ],
    verbose_feature_names_out=False,
)

import polars as pl  # or import pyarrow as pa
X_pl = pl.from_pandas(X)  # or X_pa = pa.table(X)

preprocessor.fit_transform(X_pl)
# preprocessor.set_output(transform="pandas").fit_transform(X_pl)
```

errors with
```
AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

ValueError: Specifying the columns using strings is only supported for pandas DataFrames
```

### Describe your proposed solution

scikit-learn should support those dataframes, maybe via the [python dataframe interchange protocol](https://data-apis.org/dataframe-protocol/latest/index.html).

In that regard, a new option like `set_output(transform="dataframe")` would be nice.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Some related discussion came up in #25813.

COMMENT:
> This gives us the option of "Do not extend support for other DataFrames and recommend converting DataFrames into pandas because the conversion is zero copy".

Or do we magically convert internally to pandas? If we have a full pipeline with a predictor at the end, then I don't find it too much of a hassle. If we have a `Pipeline` which is then a transformer, then we will be requested to output the same DataFrame type as what came in.

Labels: New Feature, RFC
Support other dataframes like polars and pyarrow not just pandas

### Describe the workflow you want to enable

Currently, scikit-learn nowhere claims to support [pyarrow](https://arrow.apache.org/docs/python/) or [polars](https://www.pola.rs/). And indeed,
```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler, KBinsDiscretizer
from sklearn.compose import ColumnTransformer

X, y = load_iris(as_frame=True, return_X_y=True)
sepal_cols = ["sepal length (cm)", "sepal width (cm)"]
petal_cols = ["petal length (cm)", "petal width (cm)"]

preprocessor = ColumnTransformer(
    [
        ("scaler", StandardScaler(), sepal_cols),
        ("kbin", KBinsDiscretizer(encode="ordinal"), petal_cols),
    ],
    verbose_feature_names_out=False,
)

import polars as pl  # or import pyarrow as pa
X_pl = pl.from_pandas(X)  # or X_pa = pa.table(X)

preprocessor.fit_transform(X_pl)
# preprocessor.set_output(transform="pandas").fit_transform(X_pl)
```

errors with
```
AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

ValueError: Specifying the columns using strings is only supported for pandas DataFrames
```

### Describe your proposed solution

scikit-learn should support those dataframes, maybe via the [python dataframe interchange protocol](https://data-apis.org/dataframe-protocol/latest/index.html).

In that regard, a new option like `set_output(transform="dataframe")` would be nice.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Some related discussion came up in #25813.

COMMENT:
Until https://github.com/data-apis/dataframe-api/issues/42 is decided, could we at least support the ones with `__dataframe__` (quite many already) by means of [`pandas.api.interchange.from_dataframe`](https://pandas.pydata.org/docs/reference/api/pandas.api.interchange.from_dataframe.html) (pandas v1.5.0). I would like to avoid that the users must call `X.to_pandas()`.

Or could we use https://github.com/apache/arrow-nanoarrow to support arrow arrays in general?

Labels: New Feature, RFC
Support other dataframes like polars and pyarrow not just pandas

### Describe the workflow you want to enable

Currently, scikit-learn nowhere claims to support [pyarrow](https://arrow.apache.org/docs/python/) or [polars](https://www.pola.rs/). And indeed,
```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler, KBinsDiscretizer
from sklearn.compose import ColumnTransformer

X, y = load_iris(as_frame=True, return_X_y=True)
sepal_cols = ["sepal length (cm)", "sepal width (cm)"]
petal_cols = ["petal length (cm)", "petal width (cm)"]

preprocessor = ColumnTransformer(
    [
        ("scaler", StandardScaler(), sepal_cols),
        ("kbin", KBinsDiscretizer(encode="ordinal"), petal_cols),
    ],
    verbose_feature_names_out=False,
)

import polars as pl  # or import pyarrow as pa
X_pl = pl.from_pandas(X)  # or X_pa = pa.table(X)

preprocessor.fit_transform(X_pl)
# preprocessor.set_output(transform="pandas").fit_transform(X_pl)
```

errors with
```
AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

ValueError: Specifying the columns using strings is only supported for pandas DataFrames
```

### Describe your proposed solution

scikit-learn should support those dataframes, maybe via the [python dataframe interchange protocol](https://data-apis.org/dataframe-protocol/latest/index.html).

In that regard, a new option like `set_output(transform="dataframe")` would be nice.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Some related discussion came up in #25813.

COMMENT:
> Until [data-apis/dataframe-api#42](https://github.com/data-apis/dataframe-api/issues/42) is decided, could we at least support the ones with `__dataframe__` (quite many already) by means of [`pandas.api.interchange.from_dataframe`](https://pandas.pydata.org/docs/reference/api/pandas.api.interchange.from_dataframe.html) (pandas v1.5.0). I would like to avoid that the users must call `X.to_pandas()`.

As an FYI, it looks like VegaFusion just took the interchange approach for Polars integration; consequently they got Vaex, pyarrow Tables, cuDF, _and_ Polars working with the same update, which seems like good bang for the buck 🤔 
 
https://vegafusion.io/posts/2023/2023-03-25_Release_1.1.0.html

Labels: New Feature, RFC
Support other dataframes like polars and pyarrow not just pandas

### Describe the workflow you want to enable

Currently, scikit-learn nowhere claims to support [pyarrow](https://arrow.apache.org/docs/python/) or [polars](https://www.pola.rs/). And indeed,
```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler, KBinsDiscretizer
from sklearn.compose import ColumnTransformer

X, y = load_iris(as_frame=True, return_X_y=True)
sepal_cols = ["sepal length (cm)", "sepal width (cm)"]
petal_cols = ["petal length (cm)", "petal width (cm)"]

preprocessor = ColumnTransformer(
    [
        ("scaler", StandardScaler(), sepal_cols),
        ("kbin", KBinsDiscretizer(encode="ordinal"), petal_cols),
    ],
    verbose_feature_names_out=False,
)

import polars as pl  # or import pyarrow as pa
X_pl = pl.from_pandas(X)  # or X_pa = pa.table(X)

preprocessor.fit_transform(X_pl)
# preprocessor.set_output(transform="pandas").fit_transform(X_pl)
```

errors with
```
AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

ValueError: Specifying the columns using strings is only supported for pandas DataFrames
```

### Describe your proposed solution

scikit-learn should support those dataframes, maybe via the [python dataframe interchange protocol](https://data-apis.org/dataframe-protocol/latest/index.html).

In that regard, a new option like `set_output(transform="dataframe")` would be nice.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Some related discussion came up in #25813.

COMMENT:
Now that we have more or less the infrastructure for it, we shouldn't be too shy of supporting these.

Labels: New Feature, RFC