Talip7/scikit-learn-issues-multilabel huggingface dataset:
RFC switch to Polars as the default dataframe lib in our examples

We now support polars as an output for our transformers and `ColumnTransformer`, but our examples use `pd.DataFrame`.

Our `datasets` module also returnes either a numpy array or a pandas DataFrame.

I'm suggesting that we enable datasets to return a polars dataframe, and to switch our examples from pandas to polars.

This has a few benefits:
- Polars on users' systems takes advantage of multi-core CPUs which is the case for pretty much all users these days. So it's quite faster in most cases.
- Pandas is dealing with issues related to Arrow, and even if they don't require Arrow as a required dependency, there will be behavior changes whether Arrow is installed or not at least on String Dtype.

Another thing is the (in)stability issues related to Pandas API where we need to deal with deprecation warnings very often. Although I'm not sure how stable the API is on the places where we touch the API on the polars side (maybe cc @MarcoGorelli )

WDYT @scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team 

This is not really a core part of our library, but what we put in our examples and our default choices affect people since many learn from our examples.

COMMENT:
Very interesting, indeed. Let‘s bring some structure.

#### 1. To the original question whether to switch pandas examples to polars examples
I think sooner or later we should have a mix, better document pros and cons of the different approaches, AND wait for polars 1.0.
Reason: There a many deprecations happening (eventually a good thing!) and I‘d like to have examples free of warnings.

#### 2. Making scikit-learn dataframe native
Or as @thomasjpfan puts it
> scikit-learn becomes "dataframe first" and run operations directly on dataframes

I think this better deserves its own place of discussion, e.g. an own issue.
Personal remark: It‘s something I dream of for a long time.

Labels: RFC
RFC switch to Polars as the default dataframe lib in our examples

We now support polars as an output for our transformers and `ColumnTransformer`, but our examples use `pd.DataFrame`.

Our `datasets` module also returnes either a numpy array or a pandas DataFrame.

I'm suggesting that we enable datasets to return a polars dataframe, and to switch our examples from pandas to polars.

This has a few benefits:
- Polars on users' systems takes advantage of multi-core CPUs which is the case for pretty much all users these days. So it's quite faster in most cases.
- Pandas is dealing with issues related to Arrow, and even if they don't require Arrow as a required dependency, there will be behavior changes whether Arrow is installed or not at least on String Dtype.

Another thing is the (in)stability issues related to Pandas API where we need to deal with deprecation warnings very often. Although I'm not sure how stable the API is on the places where we touch the API on the polars side (maybe cc @MarcoGorelli )

WDYT @scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team 

This is not really a core part of our library, but what we put in our examples and our default choices affect people since many learn from our examples.

COMMENT:
I totally agree that pipelines and transformers should be dataframe natives, with Numpy array being the second class.

Labels: RFC
RFC switch to Polars as the default dataframe lib in our examples

We now support polars as an output for our transformers and `ColumnTransformer`, but our examples use `pd.DataFrame`.

Our `datasets` module also returnes either a numpy array or a pandas DataFrame.

I'm suggesting that we enable datasets to return a polars dataframe, and to switch our examples from pandas to polars.

This has a few benefits:
- Polars on users' systems takes advantage of multi-core CPUs which is the case for pretty much all users these days. So it's quite faster in most cases.
- Pandas is dealing with issues related to Arrow, and even if they don't require Arrow as a required dependency, there will be behavior changes whether Arrow is installed or not at least on String Dtype.

Another thing is the (in)stability issues related to Pandas API where we need to deal with deprecation warnings very often. Although I'm not sure how stable the API is on the places where we touch the API on the polars side (maybe cc @MarcoGorelli )

WDYT @scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team 

This is not really a core part of our library, but what we put in our examples and our default choices affect people since many learn from our examples.

COMMENT:
> in between sklearn.Pipeline steps pandas might be faster, due to lack of memory copies.

Note that lack of memory copies isn't entirely true. If users enable pandas COW (which will be enabled in pandas 3.0), moving numpy data into pandas will copy that data. And mutation will copy as well. (Which I would argue is a good thing).

And if CoW isn't enabled, pandas does a lot of implicit/ defensive copies under the hood. So I think that copy will be paid down the road.

```python
import pandas as pd
pd.set_option("mode.copy_on_write", True)
rows = int(1e8)
cols = 5
x = np.ones((rows, cols))
df = pd.DataFrame(x)
```

Is double the memory off:

```python
import pandas as pd
rows = int(1e8)
cols = 5
x = np.ones((rows, cols))
df = pd.DataFrame(x)
```

Labels: RFC
RFC switch to Polars as the default dataframe lib in our examples

We now support polars as an output for our transformers and `ColumnTransformer`, but our examples use `pd.DataFrame`.

Our `datasets` module also returnes either a numpy array or a pandas DataFrame.

I'm suggesting that we enable datasets to return a polars dataframe, and to switch our examples from pandas to polars.

This has a few benefits:
- Polars on users' systems takes advantage of multi-core CPUs which is the case for pretty much all users these days. So it's quite faster in most cases.
- Pandas is dealing with issues related to Arrow, and even if they don't require Arrow as a required dependency, there will be behavior changes whether Arrow is installed or not at least on String Dtype.

Another thing is the (in)stability issues related to Pandas API where we need to deal with deprecation warnings very often. Although I'm not sure how stable the API is on the places where we touch the API on the polars side (maybe cc @MarcoGorelli )

WDYT @scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team 

This is not really a core part of our library, but what we put in our examples and our default choices affect people since many learn from our examples.

COMMENT:
Currently, we use `copy=False` when wrapping the `ndarray` to avoid the copy:

```python
import pandas as pd
import numpy as np

pd.set_option("mode.copy_on_write", True)
rows = int(1e8)
cols = 5
x = np.ones((rows, cols))
df = pd.DataFrame(x, copy=False)
```

When I run `memray` on the above, it gives the same memory usage regardless of `mode.copy_on_write`.

REF: https://github.com/scikit-learn/scikit-learn/pull/26272

Labels: RFC
RFC switch to Polars as the default dataframe lib in our examples

We now support polars as an output for our transformers and `ColumnTransformer`, but our examples use `pd.DataFrame`.

Our `datasets` module also returnes either a numpy array or a pandas DataFrame.

I'm suggesting that we enable datasets to return a polars dataframe, and to switch our examples from pandas to polars.

This has a few benefits:
- Polars on users' systems takes advantage of multi-core CPUs which is the case for pretty much all users these days. So it's quite faster in most cases.
- Pandas is dealing with issues related to Arrow, and even if they don't require Arrow as a required dependency, there will be behavior changes whether Arrow is installed or not at least on String Dtype.

Another thing is the (in)stability issues related to Pandas API where we need to deal with deprecation warnings very often. Although I'm not sure how stable the API is on the places where we touch the API on the polars side (maybe cc @MarcoGorelli )

WDYT @scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team 

This is not really a core part of our library, but what we put in our examples and our default choices affect people since many learn from our examples.

COMMENT:
+1 for supporting Polars generally.

scikit-learn definitely can be a dataframe native, but I think that some estimators will not run efficiently on most of them except if their buffers are consolidated and copied from F-order to C-order — which would come at a runtime cost we need to estimate, I think.

Labels: RFC
RFC switch to Polars as the default dataframe lib in our examples

We now support polars as an output for our transformers and `ColumnTransformer`, but our examples use `pd.DataFrame`.

Our `datasets` module also returnes either a numpy array or a pandas DataFrame.

I'm suggesting that we enable datasets to return a polars dataframe, and to switch our examples from pandas to polars.

This has a few benefits:
- Polars on users' systems takes advantage of multi-core CPUs which is the case for pretty much all users these days. So it's quite faster in most cases.
- Pandas is dealing with issues related to Arrow, and even if they don't require Arrow as a required dependency, there will be behavior changes whether Arrow is installed or not at least on String Dtype.

Another thing is the (in)stability issues related to Pandas API where we need to deal with deprecation warnings very often. Although I'm not sure how stable the API is on the places where we touch the API on the polars side (maybe cc @MarcoGorelli )

WDYT @scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team 

This is not really a core part of our library, but what we put in our examples and our default choices affect people since many learn from our examples.

COMMENT:
> Currently, we use copy=False when wrapping the ndarray to avoid the copy:

Right. That does not copy memory. And I think it is ok internally for scikit-learn IFF you regard it as move semantics. If you accept a  matrix passed by a user. You must be very sure that columns in that DataFrame are not mutated as it will actually mutate the original memory, which is still hold by a users object.

Whether that's the case in scikit-learns transformers, I don't know...

```python
import pandas as pd
pd.set_option("mode.copy_on_write", True)
rows = int(1e8)
cols = 5
x = np.ones((rows, cols))
df = pd.DataFrame(x, copy=False)

# mutate column in DataFrame
df.iloc[:, 0] = 2.0

# actually mutates the original memory
print(x)
```
```
[[2. 1. 1. 1. 1.]
 [2. 1. 1. 1. 1.]
 [2. 1. 1. 1. 1.]
 ...
 [2. 1. 1. 1. 1.]
 [2. 1. 1. 1. 1.]
 [2. 1. 1. 1. 1.]]

```

Labels: RFC
RFC switch to Polars as the default dataframe lib in our examples

We now support polars as an output for our transformers and `ColumnTransformer`, but our examples use `pd.DataFrame`.

Our `datasets` module also returnes either a numpy array or a pandas DataFrame.

I'm suggesting that we enable datasets to return a polars dataframe, and to switch our examples from pandas to polars.

This has a few benefits:
- Polars on users' systems takes advantage of multi-core CPUs which is the case for pretty much all users these days. So it's quite faster in most cases.
- Pandas is dealing with issues related to Arrow, and even if they don't require Arrow as a required dependency, there will be behavior changes whether Arrow is installed or not at least on String Dtype.

Another thing is the (in)stability issues related to Pandas API where we need to deal with deprecation warnings very often. Although I'm not sure how stable the API is on the places where we touch the API on the polars side (maybe cc @MarcoGorelli )

WDYT @scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team 

This is not really a core part of our library, but what we put in our examples and our default choices affect people since many learn from our examples.

COMMENT:
A lot of the conversations have been about memory copy. Although I enjoy the technical discussions, I want to share some of my experience in credit modelling. Again, the focus is in data, ML pipelines, because this is where dataframes are mostly used. (It is obviously very different in models. So I restrict to only pipelines.)


Transformations that are columnar: impute, scale, all algebraic combinations, clip, binning, any kind of encoding (target, woe, one-hot, rank-hot, etc.) which I argue is 80-90% of all traditional ML pipeline.

Most statistics are also columnar computations.

Non-columnar: PCA, which in my experience is very hard to put in production, and is mostly used as an exploratory tool.

I do not see any problem in copying data when doing things like PCA, unless someone is very resource constrained. I would highly recommend downsampling tech niques in that case. 

For time series, I think it is inherently columnar. For things like auto-correlation coefficient, extra data have to be generated and we can generate matrices under the hood. So no real issue.

Entropy (e.g mutual info score) related stuff is better in matrices in theory, because you will need row wise data to be used as vector in R^n, for nearest neighbor searches.. However, the default implementations of kdtrees in scipy/sklearn copies data too. I argue that these things are again mostly used in exploration, not in production. 

In conclusion, I don't think mem copy is as a big deal as it seems.

Labels: RFC
RFC switch to Polars as the default dataframe lib in our examples

We now support polars as an output for our transformers and `ColumnTransformer`, but our examples use `pd.DataFrame`.

Our `datasets` module also returnes either a numpy array or a pandas DataFrame.

I'm suggesting that we enable datasets to return a polars dataframe, and to switch our examples from pandas to polars.

This has a few benefits:
- Polars on users' systems takes advantage of multi-core CPUs which is the case for pretty much all users these days. So it's quite faster in most cases.
- Pandas is dealing with issues related to Arrow, and even if they don't require Arrow as a required dependency, there will be behavior changes whether Arrow is installed or not at least on String Dtype.

Another thing is the (in)stability issues related to Pandas API where we need to deal with deprecation warnings very often. Although I'm not sure how stable the API is on the places where we touch the API on the polars side (maybe cc @MarcoGorelli )

WDYT @scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team 

This is not really a core part of our library, but what we put in our examples and our default choices affect people since many learn from our examples.

COMMENT:
> Whether that's the case in scikit-learns transformers, I don't know...

By default, scikit-learn estimators do not write to the underlying buffer. There are common tests to make sure that read-only NumPy arrays work with every estimator. (There are a few transformers with a `copy` parameter that allow for inplace operations, but there are not many.)

Yes, I agree a good chunk of ML pipelines are columnar, so there is a strong motivation to go DataFrame first. Currently, scikit-learn is supporting:

1. NumPy arrays
2. DataFrames by converting them to NumPy arrays
3. Array API arrays (To support GPUs using PyTorch or CuPy and can be a separate code path from NumPy)

If we go DataFrame native, we will better support the DataFrame use case. Functionally, there are two ways to do this:

1. Use the [DataFrame API specification](https://github.com/data-apis/dataframe-api), when it has all the functionality we need.
2. Have a hard dependency on Polars (or Pandas) and then go through the DataFrame exchange protocol for all other DataFrames.

If I was unconstrained design-wise, I would want a hard dependency on Polars, just so I can create my own plugin expressions plugins and support polar's lazy DataFrames. Ideally, we'll have at most two code paths for every operation: "polars" and "Array API".

Labels: RFC
RFC switch to Polars as the default dataframe lib in our examples

We now support polars as an output for our transformers and `ColumnTransformer`, but our examples use `pd.DataFrame`.

Our `datasets` module also returnes either a numpy array or a pandas DataFrame.

I'm suggesting that we enable datasets to return a polars dataframe, and to switch our examples from pandas to polars.

This has a few benefits:
- Polars on users' systems takes advantage of multi-core CPUs which is the case for pretty much all users these days. So it's quite faster in most cases.
- Pandas is dealing with issues related to Arrow, and even if they don't require Arrow as a required dependency, there will be behavior changes whether Arrow is installed or not at least on String Dtype.

Another thing is the (in)stability issues related to Pandas API where we need to deal with deprecation warnings very often. Although I'm not sure how stable the API is on the places where we touch the API on the polars side (maybe cc @MarcoGorelli )

WDYT @scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team 

This is not really a core part of our library, but what we put in our examples and our default choices affect people since many learn from our examples.

COMMENT:
> If I was unconstrained design-wise, I would want a hard dependency on Polars, just so I can create my own plugin expressions plugins and support polar's lazy DataFrames.

I think this is where we are heading to in `skrub` (https://github.com/skrub-data/skrub/) taking profit that we don't have the same constraints.

Labels: RFC
RFC switch to Polars as the default dataframe lib in our examples

We now support polars as an output for our transformers and `ColumnTransformer`, but our examples use `pd.DataFrame`.

Our `datasets` module also returnes either a numpy array or a pandas DataFrame.

I'm suggesting that we enable datasets to return a polars dataframe, and to switch our examples from pandas to polars.

This has a few benefits:
- Polars on users' systems takes advantage of multi-core CPUs which is the case for pretty much all users these days. So it's quite faster in most cases.
- Pandas is dealing with issues related to Arrow, and even if they don't require Arrow as a required dependency, there will be behavior changes whether Arrow is installed or not at least on String Dtype.

Another thing is the (in)stability issues related to Pandas API where we need to deal with deprecation warnings very often. Although I'm not sure how stable the API is on the places where we touch the API on the polars side (maybe cc @MarcoGorelli )

WDYT @scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team 

This is not really a core part of our library, but what we put in our examples and our default choices affect people since many learn from our examples.

COMMENT:
> Use the [DataFrame API specification](https://github.com/data-apis/dataframe-api) when it has all the functionality we need

Realistically, this'll never happen. The spec isn't designed with Polars in mind. If you're concerned about an extra copy, you definitely wouldn't accept the compromises Polars would have to make to fit that spec.
I spent over half a year trying to turn things around, but it was an uphill battle which was affecting my mental health. Eventually I gave up.
Looks like skrub may have come to the same conclusion

> hard dependency on Polars, just so I can create my own plugin expressions plugins and support polar's lazy DataFrames. Ideally, we'll have at most two code paths for every operation: "polars" and "Array API".

I think in this thread alone there are enough people motivated enough not only to make this happen but also to maintain it long-term (but just out of interest, why would it require a hard dependency?)

Labels: RFC
RFC switch to Polars as the default dataframe lib in our examples

We now support polars as an output for our transformers and `ColumnTransformer`, but our examples use `pd.DataFrame`.

Our `datasets` module also returnes either a numpy array or a pandas DataFrame.

I'm suggesting that we enable datasets to return a polars dataframe, and to switch our examples from pandas to polars.

This has a few benefits:
- Polars on users' systems takes advantage of multi-core CPUs which is the case for pretty much all users these days. So it's quite faster in most cases.
- Pandas is dealing with issues related to Arrow, and even if they don't require Arrow as a required dependency, there will be behavior changes whether Arrow is installed or not at least on String Dtype.

Another thing is the (in)stability issues related to Pandas API where we need to deal with deprecation warnings very often. Although I'm not sure how stable the API is on the places where we touch the API on the polars side (maybe cc @MarcoGorelli )

WDYT @scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team 

This is not really a core part of our library, but what we put in our examples and our default choices affect people since many learn from our examples.

COMMENT:
> (but just out of interest, why would it require a hard dependency?)

You are right, polars can be a soft dependency. If we get a DataFrame, then the behavior could be:

- If polars is installed, then use the polars code path.
- If polars is not installed, then convert to NumPy. (Which is what we do now)

Labels: RFC