Talip7/scikit-learn-issues-multilabel huggingface dataset:
Support other dataframes like polars and pyarrow not just pandas

### Describe the workflow you want to enable

Currently, scikit-learn nowhere claims to support [pyarrow](https://arrow.apache.org/docs/python/) or [polars](https://www.pola.rs/). And indeed,
```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler, KBinsDiscretizer
from sklearn.compose import ColumnTransformer

X, y = load_iris(as_frame=True, return_X_y=True)
sepal_cols = ["sepal length (cm)", "sepal width (cm)"]
petal_cols = ["petal length (cm)", "petal width (cm)"]

preprocessor = ColumnTransformer(
    [
        ("scaler", StandardScaler(), sepal_cols),
        ("kbin", KBinsDiscretizer(encode="ordinal"), petal_cols),
    ],
    verbose_feature_names_out=False,
)

import polars as pl  # or import pyarrow as pa
X_pl = pl.from_pandas(X)  # or X_pa = pa.table(X)

preprocessor.fit_transform(X_pl)
# preprocessor.set_output(transform="pandas").fit_transform(X_pl)
```

errors with
```
AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

ValueError: Specifying the columns using strings is only supported for pandas DataFrames
```

### Describe your proposed solution

scikit-learn should support those dataframes, maybe via the [python dataframe interchange protocol](https://data-apis.org/dataframe-protocol/latest/index.html).

In that regard, a new option like `set_output(transform="dataframe")` would be nice.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Some related discussion came up in #25813.

COMMENT:
@lorentzenchr do you have some example code or link to something that shows how people use duckdb and scikt-learn now? a super quick google got me to https://duckdb.org/docs/api/python/overview.html#result-conversion which is a bit too basic(?). I'd like to see what some real world(ish) code looks like today.

Labels: New Feature, RFC
Support other dataframes like polars and pyarrow not just pandas

### Describe the workflow you want to enable

Currently, scikit-learn nowhere claims to support [pyarrow](https://arrow.apache.org/docs/python/) or [polars](https://www.pola.rs/). And indeed,
```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler, KBinsDiscretizer
from sklearn.compose import ColumnTransformer

X, y = load_iris(as_frame=True, return_X_y=True)
sepal_cols = ["sepal length (cm)", "sepal width (cm)"]
petal_cols = ["petal length (cm)", "petal width (cm)"]

preprocessor = ColumnTransformer(
    [
        ("scaler", StandardScaler(), sepal_cols),
        ("kbin", KBinsDiscretizer(encode="ordinal"), petal_cols),
    ],
    verbose_feature_names_out=False,
)

import polars as pl  # or import pyarrow as pa
X_pl = pl.from_pandas(X)  # or X_pa = pa.table(X)

preprocessor.fit_transform(X_pl)
# preprocessor.set_output(transform="pandas").fit_transform(X_pl)
```

errors with
```
AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

ValueError: Specifying the columns using strings is only supported for pandas DataFrames
```

### Describe your proposed solution

scikit-learn should support those dataframes, maybe via the [python dataframe interchange protocol](https://data-apis.org/dataframe-protocol/latest/index.html).

In that regard, a new option like `set_output(transform="dataframe")` would be nice.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Some related discussion came up in #25813.

COMMENT:
I think supporting other dataframes via `FunctionTransformer` and the like feels very much like a clever hack. For the average user it is probably way to time consuming to figure out that this is the way to make it work. It probably doesn't even cross their mind that it is possible. For me this means we should work on getting to "passing a foobar-dataframe just works".

Labels: New Feature, RFC
Support other dataframes like polars and pyarrow not just pandas

### Describe the workflow you want to enable

Currently, scikit-learn nowhere claims to support [pyarrow](https://arrow.apache.org/docs/python/) or [polars](https://www.pola.rs/). And indeed,
```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler, KBinsDiscretizer
from sklearn.compose import ColumnTransformer

X, y = load_iris(as_frame=True, return_X_y=True)
sepal_cols = ["sepal length (cm)", "sepal width (cm)"]
petal_cols = ["petal length (cm)", "petal width (cm)"]

preprocessor = ColumnTransformer(
    [
        ("scaler", StandardScaler(), sepal_cols),
        ("kbin", KBinsDiscretizer(encode="ordinal"), petal_cols),
    ],
    verbose_feature_names_out=False,
)

import polars as pl  # or import pyarrow as pa
X_pl = pl.from_pandas(X)  # or X_pa = pa.table(X)

preprocessor.fit_transform(X_pl)
# preprocessor.set_output(transform="pandas").fit_transform(X_pl)
```

errors with
```
AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

ValueError: Specifying the columns using strings is only supported for pandas DataFrames
```

### Describe your proposed solution

scikit-learn should support those dataframes, maybe via the [python dataframe interchange protocol](https://data-apis.org/dataframe-protocol/latest/index.html).

In that regard, a new option like `set_output(transform="dataframe")` would be nice.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Some related discussion came up in #25813.

COMMENT:
> Do you have some example code or link to something that shows how people use duckdb and scikt-learn now?

They simply convert to pandas before passing the data to `fit` (I can write some SQL-like data prep example if you like). This mean that they have to have pandas installed.

My personal summary:
1. I'd like to make (or better a volunteer to make) the conversion to `__dataframe__` supporting data objects magically work or clearly fail if pandas not installed.
2. Further discuss other `set_output` options.
3. I'm thinking a lot about an arrow native ML library...

Labels: New Feature, RFC
Support other dataframes like polars and pyarrow not just pandas

### Describe the workflow you want to enable

Currently, scikit-learn nowhere claims to support [pyarrow](https://arrow.apache.org/docs/python/) or [polars](https://www.pola.rs/). And indeed,
```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler, KBinsDiscretizer
from sklearn.compose import ColumnTransformer

X, y = load_iris(as_frame=True, return_X_y=True)
sepal_cols = ["sepal length (cm)", "sepal width (cm)"]
petal_cols = ["petal length (cm)", "petal width (cm)"]

preprocessor = ColumnTransformer(
    [
        ("scaler", StandardScaler(), sepal_cols),
        ("kbin", KBinsDiscretizer(encode="ordinal"), petal_cols),
    ],
    verbose_feature_names_out=False,
)

import polars as pl  # or import pyarrow as pa
X_pl = pl.from_pandas(X)  # or X_pa = pa.table(X)

preprocessor.fit_transform(X_pl)
# preprocessor.set_output(transform="pandas").fit_transform(X_pl)
```

errors with
```
AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

ValueError: Specifying the columns using strings is only supported for pandas DataFrames
```

### Describe your proposed solution

scikit-learn should support those dataframes, maybe via the [python dataframe interchange protocol](https://data-apis.org/dataframe-protocol/latest/index.html).

In that regard, a new option like `set_output(transform="dataframe")` would be nice.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Some related discussion came up in #25813.

COMMENT:
> They simply convert to pandas before passing the data to `fit` (I can write some SQL-like data prep example if you like). This mean that they have to have pandas installed.

Thanks. I wasn't sure if it was as simple as that or not. Don't think we need an example.

Labels: New Feature, RFC
Support other dataframes like polars and pyarrow not just pandas

### Describe the workflow you want to enable

Currently, scikit-learn nowhere claims to support [pyarrow](https://arrow.apache.org/docs/python/) or [polars](https://www.pola.rs/). And indeed,
```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler, KBinsDiscretizer
from sklearn.compose import ColumnTransformer

X, y = load_iris(as_frame=True, return_X_y=True)
sepal_cols = ["sepal length (cm)", "sepal width (cm)"]
petal_cols = ["petal length (cm)", "petal width (cm)"]

preprocessor = ColumnTransformer(
    [
        ("scaler", StandardScaler(), sepal_cols),
        ("kbin", KBinsDiscretizer(encode="ordinal"), petal_cols),
    ],
    verbose_feature_names_out=False,
)

import polars as pl  # or import pyarrow as pa
X_pl = pl.from_pandas(X)  # or X_pa = pa.table(X)

preprocessor.fit_transform(X_pl)
# preprocessor.set_output(transform="pandas").fit_transform(X_pl)
```

errors with
```
AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

ValueError: Specifying the columns using strings is only supported for pandas DataFrames
```

### Describe your proposed solution

scikit-learn should support those dataframes, maybe via the [python dataframe interchange protocol](https://data-apis.org/dataframe-protocol/latest/index.html).

In that regard, a new option like `set_output(transform="dataframe")` would be nice.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Some related discussion came up in #25813.

COMMENT:
I think the urgency of us supporting `polars` at the same level as supporting `pandas` just went up since pandas is voting for having pyarrow as a mandatory dependency. This increases the footprint of an environment with pandas in it substantially (pyarrow is huge). So it would be nice for us to support `polars` before that happens.

Labels: New Feature, RFC
Support other dataframes like polars and pyarrow not just pandas

### Describe the workflow you want to enable

Currently, scikit-learn nowhere claims to support [pyarrow](https://arrow.apache.org/docs/python/) or [polars](https://www.pola.rs/). And indeed,
```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler, KBinsDiscretizer
from sklearn.compose import ColumnTransformer

X, y = load_iris(as_frame=True, return_X_y=True)
sepal_cols = ["sepal length (cm)", "sepal width (cm)"]
petal_cols = ["petal length (cm)", "petal width (cm)"]

preprocessor = ColumnTransformer(
    [
        ("scaler", StandardScaler(), sepal_cols),
        ("kbin", KBinsDiscretizer(encode="ordinal"), petal_cols),
    ],
    verbose_feature_names_out=False,
)

import polars as pl  # or import pyarrow as pa
X_pl = pl.from_pandas(X)  # or X_pa = pa.table(X)

preprocessor.fit_transform(X_pl)
# preprocessor.set_output(transform="pandas").fit_transform(X_pl)
```

errors with
```
AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

ValueError: Specifying the columns using strings is only supported for pandas DataFrames
```

### Describe your proposed solution

scikit-learn should support those dataframes, maybe via the [python dataframe interchange protocol](https://data-apis.org/dataframe-protocol/latest/index.html).

In that regard, a new option like `set_output(transform="dataframe")` would be nice.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Some related discussion came up in #25813.

COMMENT:
Do you have a link to an issue or discussion about the change/schedule?

Labels: New Feature, RFC
Support other dataframes like polars and pyarrow not just pandas

### Describe the workflow you want to enable

Currently, scikit-learn nowhere claims to support [pyarrow](https://arrow.apache.org/docs/python/) or [polars](https://www.pola.rs/). And indeed,
```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler, KBinsDiscretizer
from sklearn.compose import ColumnTransformer

X, y = load_iris(as_frame=True, return_X_y=True)
sepal_cols = ["sepal length (cm)", "sepal width (cm)"]
petal_cols = ["petal length (cm)", "petal width (cm)"]

preprocessor = ColumnTransformer(
    [
        ("scaler", StandardScaler(), sepal_cols),
        ("kbin", KBinsDiscretizer(encode="ordinal"), petal_cols),
    ],
    verbose_feature_names_out=False,
)

import polars as pl  # or import pyarrow as pa
X_pl = pl.from_pandas(X)  # or X_pa = pa.table(X)

preprocessor.fit_transform(X_pl)
# preprocessor.set_output(transform="pandas").fit_transform(X_pl)
```

errors with
```
AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

ValueError: Specifying the columns using strings is only supported for pandas DataFrames
```

### Describe your proposed solution

scikit-learn should support those dataframes, maybe via the [python dataframe interchange protocol](https://data-apis.org/dataframe-protocol/latest/index.html).

In that regard, a new option like `set_output(transform="dataframe")` would be nice.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Some related discussion came up in #25813.

COMMENT:
https://github.com/pandas-dev/pandas/pull/52711

The voting period has started and it'll probably get a pass.

Labels: New Feature, RFC
Support other dataframes like polars and pyarrow not just pandas

### Describe the workflow you want to enable

Currently, scikit-learn nowhere claims to support [pyarrow](https://arrow.apache.org/docs/python/) or [polars](https://www.pola.rs/). And indeed,
```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler, KBinsDiscretizer
from sklearn.compose import ColumnTransformer

X, y = load_iris(as_frame=True, return_X_y=True)
sepal_cols = ["sepal length (cm)", "sepal width (cm)"]
petal_cols = ["petal length (cm)", "petal width (cm)"]

preprocessor = ColumnTransformer(
    [
        ("scaler", StandardScaler(), sepal_cols),
        ("kbin", KBinsDiscretizer(encode="ordinal"), petal_cols),
    ],
    verbose_feature_names_out=False,
)

import polars as pl  # or import pyarrow as pa
X_pl = pl.from_pandas(X)  # or X_pa = pa.table(X)

preprocessor.fit_transform(X_pl)
# preprocessor.set_output(transform="pandas").fit_transform(X_pl)
```

errors with
```
AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

ValueError: Specifying the columns using strings is only supported for pandas DataFrames
```

### Describe your proposed solution

scikit-learn should support those dataframes, maybe via the [python dataframe interchange protocol](https://data-apis.org/dataframe-protocol/latest/index.html).

In that regard, a new option like `set_output(transform="dataframe")` would be nice.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Some related discussion came up in #25813.

COMMENT:
> I think the urgency of us supporting `polars` at the same level as supporting `pandas` just went up since pandas is voting for having pyarrow as a mandatory dependency. This increases the footprint of an environment with pandas in it substantially (pyarrow is huge). So it would be nice for us to support `polars` before that happens.

Right, happy to give it all it takes to make this happen

The interchange protocol has been mentioned, but that would likely not be sufficient:
- there's no constructors
- there's no `to_array`-like method
- there's no way to filter on rows

This might actually be a use-case for the DataFrame Standard. I have a compat package here implements it for pandas and polars: https://github.com/data-apis/dataframe-api-compat

My immediate plans are:
- merge non-draft PRs to https://github.com/data-apis/dataframe-api
- tag the first version of the dataframe standard (2023-07, or 2023-08, depending on how long the above takes)
- update the `dataframe-api-compat` package as per the above, and try upstreaming `__dataframe_standard__` methods to pandas and polars, which (for now) would defer to the `dataframe-api-compat` package

Then, happy to help out as much as is needed to get this working in scikit-learn

Labels: New Feature, RFC
Support other dataframes like polars and pyarrow not just pandas

### Describe the workflow you want to enable

Currently, scikit-learn nowhere claims to support [pyarrow](https://arrow.apache.org/docs/python/) or [polars](https://www.pola.rs/). And indeed,
```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler, KBinsDiscretizer
from sklearn.compose import ColumnTransformer

X, y = load_iris(as_frame=True, return_X_y=True)
sepal_cols = ["sepal length (cm)", "sepal width (cm)"]
petal_cols = ["petal length (cm)", "petal width (cm)"]

preprocessor = ColumnTransformer(
    [
        ("scaler", StandardScaler(), sepal_cols),
        ("kbin", KBinsDiscretizer(encode="ordinal"), petal_cols),
    ],
    verbose_feature_names_out=False,
)

import polars as pl  # or import pyarrow as pa
X_pl = pl.from_pandas(X)  # or X_pa = pa.table(X)

preprocessor.fit_transform(X_pl)
# preprocessor.set_output(transform="pandas").fit_transform(X_pl)
```

errors with
```
AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

ValueError: Specifying the columns using strings is only supported for pandas DataFrames
```

### Describe your proposed solution

scikit-learn should support those dataframes, maybe via the [python dataframe interchange protocol](https://data-apis.org/dataframe-protocol/latest/index.html).

In that regard, a new option like `set_output(transform="dataframe")` would be nice.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Some related discussion came up in #25813.

COMMENT:
Hi, just a further note on this: I think native support for Polars is needed and it should not be bypassed with some "to_pandas" transformation. 
At the moment, Polars is: 
(1) much faster 
(2) much less memory-hungry than pandas
these two features are the main drivers for most people using it. Having to convert everything to a pandas dataframe every time to be able to interact with sklearn would severely undermine the performance and memory efficiency of the whole process, basically making the use of polars half-pointless.

Thank you for you invaluable work!

Labels: New Feature, RFC
Support other dataframes like polars and pyarrow not just pandas

### Describe the workflow you want to enable

Currently, scikit-learn nowhere claims to support [pyarrow](https://arrow.apache.org/docs/python/) or [polars](https://www.pola.rs/). And indeed,
```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler, KBinsDiscretizer
from sklearn.compose import ColumnTransformer

X, y = load_iris(as_frame=True, return_X_y=True)
sepal_cols = ["sepal length (cm)", "sepal width (cm)"]
petal_cols = ["petal length (cm)", "petal width (cm)"]

preprocessor = ColumnTransformer(
    [
        ("scaler", StandardScaler(), sepal_cols),
        ("kbin", KBinsDiscretizer(encode="ordinal"), petal_cols),
    ],
    verbose_feature_names_out=False,
)

import polars as pl  # or import pyarrow as pa
X_pl = pl.from_pandas(X)  # or X_pa = pa.table(X)

preprocessor.fit_transform(X_pl)
# preprocessor.set_output(transform="pandas").fit_transform(X_pl)
```

errors with
```
AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

ValueError: Specifying the columns using strings is only supported for pandas DataFrames
```

### Describe your proposed solution

scikit-learn should support those dataframes, maybe via the [python dataframe interchange protocol](https://data-apis.org/dataframe-protocol/latest/index.html).

In that regard, a new option like `set_output(transform="dataframe")` would be nice.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Some related discussion came up in #25813.

COMMENT:
Hi @chiaraatmudano 

My understanding is that this would proceed in two steps:
1. use a standardised `to_array_object` method, and then use the array-api. So, polars would still convert to numpy, but that's a zero-copy conversion for purely numeric data anyway (EDIT: at least, at the Series level)
2. stretch goal: use the dataframe api standard to make use of lazy computation within scikit-learn pipelines

So there wouldn't be any `to_pandas` transformation

Labels: New Feature, RFC
Support other dataframes like polars and pyarrow not just pandas

### Describe the workflow you want to enable

Currently, scikit-learn nowhere claims to support [pyarrow](https://arrow.apache.org/docs/python/) or [polars](https://www.pola.rs/). And indeed,
```python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler, KBinsDiscretizer
from sklearn.compose import ColumnTransformer

X, y = load_iris(as_frame=True, return_X_y=True)
sepal_cols = ["sepal length (cm)", "sepal width (cm)"]
petal_cols = ["petal length (cm)", "petal width (cm)"]

preprocessor = ColumnTransformer(
    [
        ("scaler", StandardScaler(), sepal_cols),
        ("kbin", KBinsDiscretizer(encode="ordinal"), petal_cols),
    ],
    verbose_feature_names_out=False,
)

import polars as pl  # or import pyarrow as pa
X_pl = pl.from_pandas(X)  # or X_pa = pa.table(X)

preprocessor.fit_transform(X_pl)
# preprocessor.set_output(transform="pandas").fit_transform(X_pl)
```

errors with
```
AttributeError: 'numpy.ndarray' object has no attribute 'columns'

During handling of the above exception, another exception occurred:

ValueError: Specifying the columns using strings is only supported for pandas DataFrames
```

### Describe your proposed solution

scikit-learn should support those dataframes, maybe via the [python dataframe interchange protocol](https://data-apis.org/dataframe-protocol/latest/index.html).

In that regard, a new option like `set_output(transform="dataframe")` would be nice.

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

Some related discussion came up in #25813.

COMMENT:
> The interchange protocol has been mentioned, but that would likely not be sufficient

Looks like I was wrong here - from the open PRs, the dataframe interchange protocol + `numpy.to_array` would indeed be enough to support Polars (at least, initial support without leveraging lazy evaluation)

Nice, this is really good to see 🎉 Feel free to ping if you'd like anything in the consortium prioritised, but it looks like things are well underway

Labels: New Feature, RFC