Talip7/scikit-learn-issues-multilabel huggingface dataset:
RFC/API (Array API) mixing devices and data types with estimators

Right now, if the user fits an estimator using a `pandas.DataFrame`, but passes a `numpy.ndarray` during `predict`, they get a warning due to missing feature names.

The situation is only to get more complicated as we're adding support for more types via array API.

Some related issues here are:
- device: data during `fit` sits on a GPU, but a CPU is used for predict (with the same data type)
- types: using one type to `fit`, and use another type during `predict`: how do we handle this both in terms of device and the type? Do we let the operator figure out if they can coerce the data into the type which can be used?
- persistence: how do we let users fit on one device, but load on another device
- estimator conversion: do we let users convert an estimator which is fit using one type/device, to an estimator compatible with another type/device?

I vaguely remember us talking about some of these issues, but I don't see any active discussion. I might have missed something.

Related: in a library like `pytorch`, you can decide which device is going to be used when you load a model's weights.

cc @thomasjpfan

COMMENT:
Note that since `_estimator_with_converted_arrays` takes a callable as argument, it provides the user with the flexibility to control device placement by leveraging the device support of the Array API:

- https://data-apis.org/array-api/latest/design_topics/device_support.html

In the future, we might want to expose a `to_device` method directly on estimators with Array API support for convenience (e.g. to move a fitted estimator from GPU to CPU or the converse without changing the Array API container types for the fitted attributes.

Labels: API, RFC, Array API
RFC/API (Array API) mixing devices and data types with estimators

Right now, if the user fits an estimator using a `pandas.DataFrame`, but passes a `numpy.ndarray` during `predict`, they get a warning due to missing feature names.

The situation is only to get more complicated as we're adding support for more types via array API.

Some related issues here are:
- device: data during `fit` sits on a GPU, but a CPU is used for predict (with the same data type)
- types: using one type to `fit`, and use another type during `predict`: how do we handle this both in terms of device and the type? Do we let the operator figure out if they can coerce the data into the type which can be used?
- persistence: how do we let users fit on one device, but load on another device
- estimator conversion: do we let users convert an estimator which is fit using one type/device, to an estimator compatible with another type/device?

I vaguely remember us talking about some of these issues, but I don't see any active discussion. I might have missed something.

Related: in a library like `pytorch`, you can decide which device is going to be used when you load a model's weights.

cc @thomasjpfan

COMMENT:
Note that it might be possible to write a generic device-aware array converter for all the array namespace implementations that support the `__dlpack__` protocol:

- https://dmlc.github.io/dlpack/latest/python_spec.html

Labels: API, RFC, Array API
RFC/API (Array API) mixing devices and data types with estimators

Right now, if the user fits an estimator using a `pandas.DataFrame`, but passes a `numpy.ndarray` during `predict`, they get a warning due to missing feature names.

The situation is only to get more complicated as we're adding support for more types via array API.

Some related issues here are:
- device: data during `fit` sits on a GPU, but a CPU is used for predict (with the same data type)
- types: using one type to `fit`, and use another type during `predict`: how do we handle this both in terms of device and the type? Do we let the operator figure out if they can coerce the data into the type which can be used?
- persistence: how do we let users fit on one device, but load on another device
- estimator conversion: do we let users convert an estimator which is fit using one type/device, to an estimator compatible with another type/device?

I vaguely remember us talking about some of these issues, but I don't see any active discussion. I might have missed something.

Related: in a library like `pytorch`, you can decide which device is going to be used when you load a model's weights.

cc @thomasjpfan

COMMENT:
I had two more thoughts about this for which it would be interesting to hear what y'all think.

1. do we even need a torch to cupy (as an example) conversion? Or would it "just work" because they are both Array API compatible?
2. maybe we should recommend using `set_config()` instead of the context manager in our docs. It might help people to not make mistakes by fitting with a context manager but predicting without and the like. At least I find it hard to predict whether fitted with context manager means I can keep using it w/o or not (if I put myself in the shoes of someone who has not looked at the code). Having the docs talk about turning it on/off as a global thing for your whole program transports a little bit the idea that this is a global flag. Of course we should also write it in words in the docs, but people read/copy&paste code snippets ...

Labels: API, RFC, Array API
RFC/API (Array API) mixing devices and data types with estimators

Right now, if the user fits an estimator using a `pandas.DataFrame`, but passes a `numpy.ndarray` during `predict`, they get a warning due to missing feature names.

The situation is only to get more complicated as we're adding support for more types via array API.

Some related issues here are:
- device: data during `fit` sits on a GPU, but a CPU is used for predict (with the same data type)
- types: using one type to `fit`, and use another type during `predict`: how do we handle this both in terms of device and the type? Do we let the operator figure out if they can coerce the data into the type which can be used?
- persistence: how do we let users fit on one device, but load on another device
- estimator conversion: do we let users convert an estimator which is fit using one type/device, to an estimator compatible with another type/device?

I vaguely remember us talking about some of these issues, but I don't see any active discussion. I might have missed something.

Related: in a library like `pytorch`, you can decide which device is going to be used when you load a model's weights.

cc @thomasjpfan

COMMENT:
> 1. do we even need a torch to cupy (as an example) conversion? Or would it "just work" because they are both Array API compatible?

Let's try. I assume you have a cuda machine at hand. If it works, let's add a non-regression test (with the appropriate double `pytest.importorskip`).

> 2. maybe we should recommend using set_config() instead of the context manager in our docs. (...)

It's probably a good idea. Let's open a PR with your suggested changes to see if it simplifies the message as expected. We can always mention that it's possible to scope the effect using the `config_context` manager as an alternative to `set_config` at then end of the page.

Labels: API, RFC, Array API
RFC/API (Array API) mixing devices and data types with estimators

Right now, if the user fits an estimator using a `pandas.DataFrame`, but passes a `numpy.ndarray` during `predict`, they get a warning due to missing feature names.

The situation is only to get more complicated as we're adding support for more types via array API.

Some related issues here are:
- device: data during `fit` sits on a GPU, but a CPU is used for predict (with the same data type)
- types: using one type to `fit`, and use another type during `predict`: how do we handle this both in terms of device and the type? Do we let the operator figure out if they can coerce the data into the type which can be used?
- persistence: how do we let users fit on one device, but load on another device
- estimator conversion: do we let users convert an estimator which is fit using one type/device, to an estimator compatible with another type/device?

I vaguely remember us talking about some of these issues, but I don't see any active discussion. I might have missed something.

Related: in a library like `pytorch`, you can decide which device is going to be used when you load a model's weights.

cc @thomasjpfan

COMMENT:
I just wanted to reference https://github.com/scikit-learn/scikit-learn/pull/29313 , as it seems there was discussion after this to raise if `fit` and `predict` are using different array namespace/device.

I also wonder if the decision should be made separately for different namespace/device and different data type?

Labels: API, RFC, Array API
RFC/API (Array API) mixing devices and data types with estimators

Right now, if the user fits an estimator using a `pandas.DataFrame`, but passes a `numpy.ndarray` during `predict`, they get a warning due to missing feature names.

The situation is only to get more complicated as we're adding support for more types via array API.

Some related issues here are:
- device: data during `fit` sits on a GPU, but a CPU is used for predict (with the same data type)
- types: using one type to `fit`, and use another type during `predict`: how do we handle this both in terms of device and the type? Do we let the operator figure out if they can coerce the data into the type which can be used?
- persistence: how do we let users fit on one device, but load on another device
- estimator conversion: do we let users convert an estimator which is fit using one type/device, to an estimator compatible with another type/device?

I vaguely remember us talking about some of these issues, but I don't see any active discussion. I might have missed something.

Related: in a library like `pytorch`, you can decide which device is going to be used when you load a model's weights.

cc @thomasjpfan

COMMENT:
People are happy to have expensive hardware to train fast, but want to have cheap hardware to predict, and that means we need to be able to "move" across devices.

I don't think we have to support all combinations to start with, but having utility functions such as `sklearn.convert_to_type` which takes an object (Estimator) and tries to do the conversion makes sense. It also makes sense for it to fail on certain cases, with a nice message about what kind of conversion should work.

Labels: API, RFC, Array API
RFC/API (Array API) mixing devices and data types with estimators

Right now, if the user fits an estimator using a `pandas.DataFrame`, but passes a `numpy.ndarray` during `predict`, they get a warning due to missing feature names.

The situation is only to get more complicated as we're adding support for more types via array API.

Some related issues here are:
- device: data during `fit` sits on a GPU, but a CPU is used for predict (with the same data type)
- types: using one type to `fit`, and use another type during `predict`: how do we handle this both in terms of device and the type? Do we let the operator figure out if they can coerce the data into the type which can be used?
- persistence: how do we let users fit on one device, but load on another device
- estimator conversion: do we let users convert an estimator which is fit using one type/device, to an estimator compatible with another type/device?

I vaguely remember us talking about some of these issues, but I don't see any active discussion. I might have missed something.

Related: in a library like `pytorch`, you can decide which device is going to be used when you load a model's weights.

cc @thomasjpfan

COMMENT:
> sklearn.convert_to_type which takes an object (Estimator) and tries to do the conversion makes sense.

Reading over https://github.com/scikit-learn/scikit-learn/pull/29313, it seems we stalled because ideally the function would be recursive (i.e. deal with estimators inside pipelines). Naive question, may not be relevant - would metadata routing be useful for this?

Labels: API, RFC, Array API
RFC/API (Array API) mixing devices and data types with estimators

Right now, if the user fits an estimator using a `pandas.DataFrame`, but passes a `numpy.ndarray` during `predict`, they get a warning due to missing feature names.

The situation is only to get more complicated as we're adding support for more types via array API.

Some related issues here are:
- device: data during `fit` sits on a GPU, but a CPU is used for predict (with the same data type)
- types: using one type to `fit`, and use another type during `predict`: how do we handle this both in terms of device and the type? Do we let the operator figure out if they can coerce the data into the type which can be used?
- persistence: how do we let users fit on one device, but load on another device
- estimator conversion: do we let users convert an estimator which is fit using one type/device, to an estimator compatible with another type/device?

I vaguely remember us talking about some of these issues, but I don't see any active discussion. I might have missed something.

Related: in a library like `pytorch`, you can decide which device is going to be used when you load a model's weights.

cc @thomasjpfan

COMMENT:
I don't see how metadata routing could play a role here.

And yes, it would be recursive, the same way our `clone` is recursive. I don't think that should be a blocker for us.

Labels: API, RFC, Array API
Use `np.uint8` as default dtype for `OneHotEncoder` instead of `np.float64`

### Describe the workflow you want to enable

`sklearn.preprocessing.OneHotEncoder` should use as `dtype` the `np.uint8` by default instead of `np.float64` as it is currently. I don't see the reasoning behind using `np.float64`, this if anything only causes memory explosions and potentially (?) even slows down the computation.

### Describe your proposed solution

Make `np.uint8` as the default `dtype` parameter of `sklearn.preprocessing.OneHotEncoder`

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
I would like to take this issue. Thanks

Edit - I change the ``dtype`` from ``np.float64`` to ``np.uint8`` for ``sklearn.preprocessing.OneHotEncoder``. Can you review my PR #26068.

Labels: New Feature, module:preprocessing
Use `np.uint8` as default dtype for `OneHotEncoder` instead of `np.float64`

### Describe the workflow you want to enable

`sklearn.preprocessing.OneHotEncoder` should use as `dtype` the `np.uint8` by default instead of `np.float64` as it is currently. I don't see the reasoning behind using `np.float64`, this if anything only causes memory explosions and potentially (?) even slows down the computation.

### Describe your proposed solution

Make `np.uint8` as the default `dtype` parameter of `sklearn.preprocessing.OneHotEncoder`

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
For backwards compatibility, we would need a [deprecation cycle](https://scikit-learn.org/dev/developers/contributing.html#deprecation) before changing the default. There are two use cases I see:

1. I suspect the output is float because most of our estimators will convert it's input to a `float64`. If `OneHotEncoder`  outputs an integer array and is a part of a `Pipeline` then the integer array will be passed to an estimator. That estimator will likely convert the integer array into a floating array which is a memory copy. With `OneHotEncoder`'s default `float` output, there is no memory copy taking place.
2. That being said, `OneHotEncoder` is usually in a `ColumnTransformer`, and the `ColumnTransformer` will make a copy because it has to run `np.hstack` or `pd.concat` on each transformer. The copy happens regardless of the output dtype of `OneHotEncoder`.

For case 1, changing the default to int will create a memory copy, but I think it is not very common to **only** have a `OneHotEnoder` in a pipeline.

For case 2, changing the default to int will use less memory and the number of memory copies are the same.

In summary, I think changing to default to an integer is okay and worth going through a deprecation.

Labels: New Feature, module:preprocessing
Use `np.uint8` as default dtype for `OneHotEncoder` instead of `np.float64`

### Describe the workflow you want to enable

`sklearn.preprocessing.OneHotEncoder` should use as `dtype` the `np.uint8` by default instead of `np.float64` as it is currently. I don't see the reasoning behind using `np.float64`, this if anything only causes memory explosions and potentially (?) even slows down the computation.

### Describe your proposed solution

Make `np.uint8` as the default `dtype` parameter of `sklearn.preprocessing.OneHotEncoder`

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
@thomasjpfan I am not convinced this is worth the annoyance of the deprecation where everyone must set the `dtype` during 2 versions. I agree that integers would be more efficient, but if you get an out-of-memory error, you can always set the `dtype` to go around.

If we go this path then, we should also consider modifying the default of `OrdinalEncoder`.

Labels: New Feature, module:preprocessing
Use `np.uint8` as default dtype for `OneHotEncoder` instead of `np.float64`

### Describe the workflow you want to enable

`sklearn.preprocessing.OneHotEncoder` should use as `dtype` the `np.uint8` by default instead of `np.float64` as it is currently. I don't see the reasoning behind using `np.float64`, this if anything only causes memory explosions and potentially (?) even slows down the computation.

### Describe your proposed solution

Make `np.uint8` as the default `dtype` parameter of `sklearn.preprocessing.OneHotEncoder`

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> @thomasjpfan I am not convinced this is worth the annoyance of the deprecation where everyone must set the `dtype` during 2 versions. I agree that integers would be more efficient, but if you get an out-of-memory error, you can always set the `dtype` to go around.
> 
> If we go this path then, we should also consider modifying the default of `OrdinalEncoder`.

The `OrdinalEncoder` does not present much of an issue, though, as that's not binary encoding and you could forgive it the memory wastage (if any) as `np.int32` or even `np.int64` would be used there anyway. This is not the case with the `OneHotEncoder` however, where the memory explosion (and wastage) when working with large datasets is significant and completely unnecessary.

Labels: New Feature, module:preprocessing
Use `np.uint8` as default dtype for `OneHotEncoder` instead of `np.float64`

### Describe the workflow you want to enable

`sklearn.preprocessing.OneHotEncoder` should use as `dtype` the `np.uint8` by default instead of `np.float64` as it is currently. I don't see the reasoning behind using `np.float64`, this if anything only causes memory explosions and potentially (?) even slows down the computation.

### Describe your proposed solution

Make `np.uint8` as the default `dtype` parameter of `sklearn.preprocessing.OneHotEncoder`

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
Just to make it explicit: it is currently possible to `OneHotEncoder(dtype=np.uint8, ...)` for users who run into memory size problems. Another parameter relevant to the amount of memory used is `sparse_output`, which is set to `True` by default.

If it is possible to stick with a sparse matrix the difference between float and int should be less, but I'm not sure what happens in Thomas' case (2).

Has someone done a benchmark looking at how much memory would actually be saved for some (reasonably sized) datasets?

To me it feels like we have a good option for the rare(?) users that have very large outputs from `OneHotEncoder`, and for the more common(?) user who is far away from memory limits and probably in case (2) it doesn't really matter. This makes me lean towards "no change needed".

Labels: New Feature, module:preprocessing
Use `np.uint8` as default dtype for `OneHotEncoder` instead of `np.float64`

### Describe the workflow you want to enable

`sklearn.preprocessing.OneHotEncoder` should use as `dtype` the `np.uint8` by default instead of `np.float64` as it is currently. I don't see the reasoning behind using `np.float64`, this if anything only causes memory explosions and potentially (?) even slows down the computation.

### Describe your proposed solution

Make `np.uint8` as the default `dtype` parameter of `sklearn.preprocessing.OneHotEncoder`

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
> Just to make it explicit: it is currently possible to `OneHotEncoder(dtype=np.uint8, ...)` for users who run into memory size problems. Another parameter relevant to the amount of memory used is `sparse_output`, which is set to `True` by default.
> 
> If it is possible to stick with a sparse matrix the difference between float and int should be less, but I'm not sure what happens in Thomas' case (2).
> 
> Has someone done a benchmark looking at how much memory would actually be saved for some (reasonably sized) datasets?
> 
> To me it feels like we have a good option for the rare(?) users that have very large outputs from `OneHotEncoder`, and for the more common(?) user who is far away from memory limits and probably in case (2) it doesn't really matter. This makes me lean towards "no change needed".

I have found in my use case that doing the single change (changing `dtype` to `np.uint8`) cut my RAM usage by about 40 GB. (which represented about 40% of my total memory usage) Now, my dataset is large, and I use a lot of multiprocessing, but it is important to remember (I found) that even with something like `GridSearchCV` with having `n_job`s bigger than one, this memory requirement difference will be further multiplied by each process, ultimately resulting in a hugely significant memory requirement difference to when `dtype=np.uint8` is used.

Labels: New Feature, module:preprocessing
Use `np.uint8` as default dtype for `OneHotEncoder` instead of `np.float64`

### Describe the workflow you want to enable

`sklearn.preprocessing.OneHotEncoder` should use as `dtype` the `np.uint8` by default instead of `np.float64` as it is currently. I don't see the reasoning behind using `np.float64`, this if anything only causes memory explosions and potentially (?) even slows down the computation.

### Describe your proposed solution

Make `np.uint8` as the default `dtype` parameter of `sklearn.preprocessing.OneHotEncoder`

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
If we decide that we should standardize on a `output_dtype` constructor parameter for several preprocessing transformers (as is being discussed in #25845), the we could use a deprecation cycle to do both changes (the renaming and the change of default value to `np.uint8`) at the same time.

Labels: New Feature, module:preprocessing
Use `np.uint8` as default dtype for `OneHotEncoder` instead of `np.float64`

### Describe the workflow you want to enable

`sklearn.preprocessing.OneHotEncoder` should use as `dtype` the `np.uint8` by default instead of `np.float64` as it is currently. I don't see the reasoning behind using `np.float64`, this if anything only causes memory explosions and potentially (?) even slows down the computation.

### Describe your proposed solution

Make `np.uint8` as the default `dtype` parameter of `sklearn.preprocessing.OneHotEncoder`

### Describe alternatives you've considered, if relevant

_No response_

### Additional context

_No response_

COMMENT:
If not, then we could do the change of default value as a lightweight breaking change (without an annoying deprecation cycle) targeting a still hypothetical scikit-learn 2.0.

Labels: New Feature, module:preprocessing