Talip7/scikit-learn-issues-multilabel huggingface dataset:
RFC switch to Polars as the default dataframe lib in our examples

We now support polars as an output for our transformers and `ColumnTransformer`, but our examples use `pd.DataFrame`.

Our `datasets` module also returnes either a numpy array or a pandas DataFrame.

I'm suggesting that we enable datasets to return a polars dataframe, and to switch our examples from pandas to polars.

This has a few benefits:
- Polars on users' systems takes advantage of multi-core CPUs which is the case for pretty much all users these days. So it's quite faster in most cases.
- Pandas is dealing with issues related to Arrow, and even if they don't require Arrow as a required dependency, there will be behavior changes whether Arrow is installed or not at least on String Dtype.

Another thing is the (in)stability issues related to Pandas API where we need to deal with deprecation warnings very often. Although I'm not sure how stable the API is on the places where we touch the API on the polars side (maybe cc @MarcoGorelli )

WDYT @scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team 

This is not really a core part of our library, but what we put in our examples and our default choices affect people since many learn from our examples.

COMMENT:
I also agree with Gael. I think that switching some examples with
documentation that both pandas and polars are supported is a good idea.

On Thu, 1 Feb 2024 at 14:09, Guillaume Lemaitre ***@***.***>
wrote:

> I agree with @GaelVaroquaux <https://github.com/GaelVaroquaux>. I would
> be considering having a mix of example, even more when we don't take profit
> from polars features.
>
> Currently our support for polars is closer to rewrapping a NumPy arrays
> into a Polars DataFrame without leveraging the lazyness.
>
> In an ideal world, if we come to have estimator to leverage those features
> (e.g. via the dataframe API) then we should certainly demonstrate it in the
> documentation.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/scikit-learn/scikit-learn/issues/28341#issuecomment-1921291070>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEB3BIX3BT42R52HJ2GBOJTYROHZ3AVCNFSM6AAAAABCUYK3X6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRRGI4TCMBXGA>
> .
> You are receiving this because you are on a team that was mentioned.Message
> ID: ***@***.***>
>

Labels: RFC
RFC switch to Polars as the default dataframe lib in our examples

We now support polars as an output for our transformers and `ColumnTransformer`, but our examples use `pd.DataFrame`.

Our `datasets` module also returnes either a numpy array or a pandas DataFrame.

I'm suggesting that we enable datasets to return a polars dataframe, and to switch our examples from pandas to polars.

This has a few benefits:
- Polars on users' systems takes advantage of multi-core CPUs which is the case for pretty much all users these days. So it's quite faster in most cases.
- Pandas is dealing with issues related to Arrow, and even if they don't require Arrow as a required dependency, there will be behavior changes whether Arrow is installed or not at least on String Dtype.

Another thing is the (in)stability issues related to Pandas API where we need to deal with deprecation warnings very often. Although I'm not sure how stable the API is on the places where we touch the API on the polars side (maybe cc @MarcoGorelli )

WDYT @scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team 

This is not really a core part of our library, but what we put in our examples and our default choices affect people since many learn from our examples.

COMMENT:
> Although I'm not sure how stable the API is on the places where we touch the API on the polars side

As far as I can tell, internally you're just using `__dataframe__` (the dataframe interchange protocol) and `__array__`:
- `__dataframe__`: this is implemented natively without relying on pyarrow (thanks to a lot of work by the incredible @stinodego), and its API should be stable. Even if there's any internal change in Polars, the public API in `__dataframe__` shouldn't change (that's one good thing about standards!)
- `__array__`: for numeric and string datatypes, pyarrow isn't required (though it is required for Categorical). Pretty sure this one can be considered stable anyway

As for the API you'd use in the examples - there is a chance for deprecations to show up there, but there's also a well-written deprecation policy https://docs.pola.rs/development/versioning/#deprecation-period (note: the part at the end about breaking releases will be updated for the upcoming 1.0 release)

> In an ideal world, if we come to have estimator to leverage those features (e.g. via the dataframe API)

I've been having a hard time convincing the relevant people that those (lazy) features are a "must-have", so no promises that this will be possible I'm afraid. Input on https://github.com/data-apis/dataframe-api/issues/346 would be appreciated (whether you agree, or whether I've misunderstood and am wrong)

Labels: RFC
RFC switch to Polars as the default dataframe lib in our examples

We now support polars as an output for our transformers and `ColumnTransformer`, but our examples use `pd.DataFrame`.

Our `datasets` module also returnes either a numpy array or a pandas DataFrame.

I'm suggesting that we enable datasets to return a polars dataframe, and to switch our examples from pandas to polars.

This has a few benefits:
- Polars on users' systems takes advantage of multi-core CPUs which is the case for pretty much all users these days. So it's quite faster in most cases.
- Pandas is dealing with issues related to Arrow, and even if they don't require Arrow as a required dependency, there will be behavior changes whether Arrow is installed or not at least on String Dtype.

Another thing is the (in)stability issues related to Pandas API where we need to deal with deprecation warnings very often. Although I'm not sure how stable the API is on the places where we touch the API on the polars side (maybe cc @MarcoGorelli )

WDYT @scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team 

This is not really a core part of our library, but what we put in our examples and our default choices affect people since many learn from our examples.

COMMENT:
Panda's block manager that supports 2D arrays is still nicer memory-wise when it comes to the roundtrip: `numpy` -> `dataframe` -> `numpy`. Pandas can hold on to numerical F-contiguous or C-contiguous NumPy array underneath the covers. This allows us to use Panda's dataframes in a scikit-learn `Pipeline` without introducing memory overhead.

I do not think polars is as a point where it can do the roundtrip without making a copy, which increasess memory usage. Because of this memory issue, I would still recommend converting polars dataframes to pandas before handing it over to scikit-learn's `Pipeline`.

With the memory issue in polars, I do not think we are in a good state to promote it's usage. I see two options to improve the situation around polars:

- polars stores 2d NumPy arrays without converting it into a columnar format
- scikit-learn becomes "dataframe first" and run operations directly on dataframes
	- We can make use of lazy dataframes in this case.
	- Not all algorithms are suitable for columnar data. (anything with `matmul` we need to be rethought)

Labels: RFC
RFC switch to Polars as the default dataframe lib in our examples

We now support polars as an output for our transformers and `ColumnTransformer`, but our examples use `pd.DataFrame`.

Our `datasets` module also returnes either a numpy array or a pandas DataFrame.

I'm suggesting that we enable datasets to return a polars dataframe, and to switch our examples from pandas to polars.

This has a few benefits:
- Polars on users' systems takes advantage of multi-core CPUs which is the case for pretty much all users these days. So it's quite faster in most cases.
- Pandas is dealing with issues related to Arrow, and even if they don't require Arrow as a required dependency, there will be behavior changes whether Arrow is installed or not at least on String Dtype.

Another thing is the (in)stability issues related to Pandas API where we need to deal with deprecation warnings very often. Although I'm not sure how stable the API is on the places where we touch the API on the polars side (maybe cc @MarcoGorelli )

WDYT @scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team 

This is not really a core part of our library, but what we put in our examples and our default choices affect people since many learn from our examples.

COMMENT:
I was mentioning the same memory issue, but @thomasjpfan already did that better just now

> * Pandas is dealing with issues related to Arrow, and even if they don't require Arrow as a required dependency, there will be behavior changes whether Arrow is installed or not at least on String Dtype.

FWIW I don't expect any behaviour changes for the string dtype (you could also already test that with the latest pandas, which reminds me I should do a scikit-learn test run with pandas Copy-on-Write enabled). When converting to numpy, this still becomes object dtype, and I assume that's mostly the way that scikit-learn encounters it eventually.

(and nitpick: it's PyArrow, not "Arrow" in general, there are many Arrow implementations, including one embedded in polars ;))

Labels: RFC
RFC switch to Polars as the default dataframe lib in our examples

We now support polars as an output for our transformers and `ColumnTransformer`, but our examples use `pd.DataFrame`.

Our `datasets` module also returnes either a numpy array or a pandas DataFrame.

I'm suggesting that we enable datasets to return a polars dataframe, and to switch our examples from pandas to polars.

This has a few benefits:
- Polars on users' systems takes advantage of multi-core CPUs which is the case for pretty much all users these days. So it's quite faster in most cases.
- Pandas is dealing with issues related to Arrow, and even if they don't require Arrow as a required dependency, there will be behavior changes whether Arrow is installed or not at least on String Dtype.

Another thing is the (in)stability issues related to Pandas API where we need to deal with deprecation warnings very often. Although I'm not sure how stable the API is on the places where we touch the API on the polars side (maybe cc @MarcoGorelli )

WDYT @scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team 

This is not really a core part of our library, but what we put in our examples and our default choices affect people since many learn from our examples.

COMMENT:
> I should do a scikit-learn test run with pandas Copy-on-Write enabled

FYI I did a quick and dirty attempt https://github.com/scikit-learn/scikit-learn/issues/27879#issuecomment-1841266926. The preliminary result is that there are 2 failures but some of the issues may not be caught by our tests e.g. the one reported in https://github.com/scikit-learn/scikit-learn/issues/27879.

Labels: RFC
RFC switch to Polars as the default dataframe lib in our examples

We now support polars as an output for our transformers and `ColumnTransformer`, but our examples use `pd.DataFrame`.

Our `datasets` module also returnes either a numpy array or a pandas DataFrame.

I'm suggesting that we enable datasets to return a polars dataframe, and to switch our examples from pandas to polars.

This has a few benefits:
- Polars on users' systems takes advantage of multi-core CPUs which is the case for pretty much all users these days. So it's quite faster in most cases.
- Pandas is dealing with issues related to Arrow, and even if they don't require Arrow as a required dependency, there will be behavior changes whether Arrow is installed or not at least on String Dtype.

Another thing is the (in)stability issues related to Pandas API where we need to deal with deprecation warnings very often. Although I'm not sure how stable the API is on the places where we touch the API on the polars side (maybe cc @MarcoGorelli )

WDYT @scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team 

This is not really a core part of our library, but what we put in our examples and our default choices affect people since many learn from our examples.

COMMENT:
Hi @thomasjpfan - if the
- preprocess in Polars
- call `.to_pandas()` before passing to a scikit-learn Pipeline

workflow is the most memory-efficient one, then that might be a good one to show in examples?

Labels: RFC
RFC switch to Polars as the default dataframe lib in our examples

We now support polars as an output for our transformers and `ColumnTransformer`, but our examples use `pd.DataFrame`.

Our `datasets` module also returnes either a numpy array or a pandas DataFrame.

I'm suggesting that we enable datasets to return a polars dataframe, and to switch our examples from pandas to polars.

This has a few benefits:
- Polars on users' systems takes advantage of multi-core CPUs which is the case for pretty much all users these days. So it's quite faster in most cases.
- Pandas is dealing with issues related to Arrow, and even if they don't require Arrow as a required dependency, there will be behavior changes whether Arrow is installed or not at least on String Dtype.

Another thing is the (in)stability issues related to Pandas API where we need to deal with deprecation warnings very often. Although I'm not sure how stable the API is on the places where we touch the API on the polars side (maybe cc @MarcoGorelli )

WDYT @scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team 

This is not really a core part of our library, but what we put in our examples and our default choices affect people since many learn from our examples.

COMMENT:
@MarcoGorelli 

> - preprocess in Polars
> - call .to_pandas() before passing to a scikit-learn Pipeline

There are two types of examples:

- If the preprocessing in polar does not have state, then that is a simple example.
- If preprocessing has state which is learned during `fit` and used during `transform`, then the example is to wrap the Polars expressions into a scikit-learn transformer.

@abstractqqq scikit-learn's transformers is mostly about keeping track of state between `fit` and `transform`. For example:

- We can write a Polars expression to do `SimpleScalar`, but it'll need some state object to hold on to the mean and std during `fit` time. In this case, the state object can easily be another Polars DataFrame. During `transform`, there needs to be another Polars expression that takes the `mean` and `std` and uses it on new data. (During `transform` time, we do not need to compute the mean anymore).
- For imputers, we need to store a statistic for the training data and use it to impute the test data.

Due to the constraints of scikit-learn, I think it's better to look into Polars for ML in a separate project as you have done with `polars-ds`. There is a good subset of ML techniques and algorithms that can make use of Polar's plugin system, and it is worth exploring.

Labels: RFC
RFC switch to Polars as the default dataframe lib in our examples

We now support polars as an output for our transformers and `ColumnTransformer`, but our examples use `pd.DataFrame`.

Our `datasets` module also returnes either a numpy array or a pandas DataFrame.

I'm suggesting that we enable datasets to return a polars dataframe, and to switch our examples from pandas to polars.

This has a few benefits:
- Polars on users' systems takes advantage of multi-core CPUs which is the case for pretty much all users these days. So it's quite faster in most cases.
- Pandas is dealing with issues related to Arrow, and even if they don't require Arrow as a required dependency, there will be behavior changes whether Arrow is installed or not at least on String Dtype.

Another thing is the (in)stability issues related to Pandas API where we need to deal with deprecation warnings very often. Although I'm not sure how stable the API is on the places where we touch the API on the polars side (maybe cc @MarcoGorelli )

WDYT @scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team 

This is not really a core part of our library, but what we put in our examples and our default choices affect people since many learn from our examples.

COMMENT:
@thomasjpfan You are exactly right it is about keeping the states. But I think it boils down to two steps: (1) learn from train. We can learn the mean, std, from a Polars df, and this is multithreaded by default in Polars. (2) we can then pass these learned knowledge into expressions in Polars. E.g pl.col("a").fill_null(learned_value). Now this is a perfectly valid way of preserving states. Once we have a list of such expressions, we transform each at once, which again will be in parallel in Polars by default

Labels: RFC
RFC switch to Polars as the default dataframe lib in our examples

We now support polars as an output for our transformers and `ColumnTransformer`, but our examples use `pd.DataFrame`.

Our `datasets` module also returnes either a numpy array or a pandas DataFrame.

I'm suggesting that we enable datasets to return a polars dataframe, and to switch our examples from pandas to polars.

This has a few benefits:
- Polars on users' systems takes advantage of multi-core CPUs which is the case for pretty much all users these days. So it's quite faster in most cases.
- Pandas is dealing with issues related to Arrow, and even if they don't require Arrow as a required dependency, there will be behavior changes whether Arrow is installed or not at least on String Dtype.

Another thing is the (in)stability issues related to Pandas API where we need to deal with deprecation warnings very often. Although I'm not sure how stable the API is on the places where we touch the API on the polars side (maybe cc @MarcoGorelli )

WDYT @scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team 

This is not really a core part of our library, but what we put in our examples and our default choices affect people since many learn from our examples.

COMMENT:
> FWIW I don't expect any behaviour changes for the string dtype (you could also already test that with the latest pandas

@jorisvandenbossche [this](https://github.com/pandas-dev/pandas/issues/54466#issuecomment-1914653183) and [this](https://github.com/pandas-dev/pandas/issues/54466#issuecomment-1914826310) are where my concerns come from.

Context:

> Trying to simply install pyarrow to silence the DeprecationWarning causes our tests to fail
> ...

I don't know the details and if they're relevant here, but it could be.

Labels: RFC
RFC switch to Polars as the default dataframe lib in our examples

We now support polars as an output for our transformers and `ColumnTransformer`, but our examples use `pd.DataFrame`.

Our `datasets` module also returnes either a numpy array or a pandas DataFrame.

I'm suggesting that we enable datasets to return a polars dataframe, and to switch our examples from pandas to polars.

This has a few benefits:
- Polars on users' systems takes advantage of multi-core CPUs which is the case for pretty much all users these days. So it's quite faster in most cases.
- Pandas is dealing with issues related to Arrow, and even if they don't require Arrow as a required dependency, there will be behavior changes whether Arrow is installed or not at least on String Dtype.

Another thing is the (in)stability issues related to Pandas API where we need to deal with deprecation warnings very often. Although I'm not sure how stable the API is on the places where we touch the API on the polars side (maybe cc @MarcoGorelli )

WDYT @scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team 

This is not really a core part of our library, but what we put in our examples and our default choices affect people since many learn from our examples.

COMMENT:
Whatever the conclusions, this peeks my interest. ^^

Let's make clear what we can do on the Polars side.

- Zero copy from a 2D numpy matrix to a Polars `DataFrame` is possible in depending on the row-order. We can memory map the columns.

- Zero copy from a Polars `DataFrame` to to numpy matrix will not be possible. However, we heavily parallelized the copy, and don't expect it to be the bottleneck. I expect there are quite some implicit copies in a current implementations.

- 1D columnar (without nulls) will be completely zero-copy in both directions.

- 2D numpy matrix (without nulls) will be completely zero-copy to the Polars `Array` data-type and back.

We will implement [the numpy array interface](https://github.com/pola-rs/polars/issues/13980) very soon and that will follow the above rules on (zero)-copyness.

Labels: RFC
RFC switch to Polars as the default dataframe lib in our examples

We now support polars as an output for our transformers and `ColumnTransformer`, but our examples use `pd.DataFrame`.

Our `datasets` module also returnes either a numpy array or a pandas DataFrame.

I'm suggesting that we enable datasets to return a polars dataframe, and to switch our examples from pandas to polars.

This has a few benefits:
- Polars on users' systems takes advantage of multi-core CPUs which is the case for pretty much all users these days. So it's quite faster in most cases.
- Pandas is dealing with issues related to Arrow, and even if they don't require Arrow as a required dependency, there will be behavior changes whether Arrow is installed or not at least on String Dtype.

Another thing is the (in)stability issues related to Pandas API where we need to deal with deprecation warnings very often. Although I'm not sure how stable the API is on the places where we touch the API on the polars side (maybe cc @MarcoGorelli )

WDYT @scikit-learn/core-devs @scikit-learn/documentation-team @scikit-learn/contributor-experience-team 

This is not really a core part of our library, but what we put in our examples and our default choices affect people since many learn from our examples.

COMMENT:
So it seems a [to be confirmed with benchmarks] summary of this thread is:

- pre/post `sklearn` steps might be / most probably are quite faster on `polars`
- in between `sklearn.Pipeline` steps `pandas` might be faster, due to lack of memory copies.

I'm not sure, and I would need to see benchmarks to confirm, but my gut feeling is that in a casual data/ml pipeline a fair amount of process time is spent on preprocessing the data before getting to the first step of a scikit-learn pipeline in all that data cleaning and manipulation steps. And if those steps can benefit from switching to polars to the point that the overhead spent in sklearn part is compensated for, then it's worth switching.

Now even if that statement is true, a fundamental question would be: should we as sklearn care about that? Since users can independently choose what to do before and after sklearn, a jerk reaction answer might be no, but at the same time, it would feel weird if I go to teach people how to deal with data, and use polars, and then have only pandas on our examples. I really don't know.

One thing that I personally feel, is that we have a position to have an influence in ML engineers' choices, and that means we should think in the context of a whole workflow rather than sklearn only. But I'm also not completely set on whether that's the right position to have.

This has been a great thread, and I think to further the discussion, we need more data to make better informed decisions. I'll try to get the data. But it will take a while since I'd need a good chunk of time to focus on this. Checking the memory / CPU footprint of our examples would be a start.

Labels: RFC