RFC/API (Array API) mixing devices and data types with estimators

Right now, if the user fits an estimator using a `pandas.DataFrame`, but passes a `numpy.ndarray` during `predict`, they get a warning due to missing feature names.

The situation is only to get more complicated as we're adding support for more types via array API.

Some related issues here are:
- device: data during `fit` sits on a GPU, but a CPU is used for predict (with the same data type)
- types: using one type to `fit`, and use another type during `predict`: how do we handle this both in terms of device and the type? Do we let the operator figure out if they can coerce the data into the type which can be used?
- persistence: how do we let users fit on one device, but load on another device
- estimator conversion: do we let users convert an estimator which is fit using one type/device, to an estimator compatible with another type/device?

I vaguely remember us talking about some of these issues, but I don't see any active discussion. I might have missed something.

Related: in a library like `pytorch`, you can decide which device is going to be used when you load a model's weights.

cc @thomasjpfan

COMMENT:
The current state of thinking regarding estimator attributes and their conversion is in the documentation https://scikit-learn.org/dev/modules/array_api.html. TL;DR: the user has to convert it explicitly.

A reason not to offer something more "built in" is that conversion from array library X to Numpy and the other direction is something that depends on the array library. For Cupy to Numpy you need `cp_arr.get()`, for pytorch to numpy you need something like `torch_arr.cpu().numpy()`, etc. And to offer something like array lib X to array lib Y conversion you quickly end up with a combinatorial explosion of possibilities.

This means if you want to fit with one array library and predict with another you will have to convert your model's attributes.

You'd have to try out what happens with your array library of choice regarding "fit with data on a GPU" and "predict with data on a CPU". For example, PyTorch will raise an exception if you attempt to mix and match GPU and CPU (haven't tested what happens if you mix and match GPU1 and GPU2).

I think it makes sense to ask for explicit action from the user because of the performance costs of moving data from one device to the next. It seems like the chances are high that a mistake happened if you fit a model with a GPU and then predict with CPU without being explicit about it. For example I can imagine that a frequent case would be fitting on a GPU but then in production predicting on single samples with a CPU. Swapping from development/fitting to production is such a big step that requires lots of explicit "we are now in production!!" that I think we'd be in good company by also making things explicit.

Conversion when persisting a model or when loading it sounds like a good feature for a library that makes it easy to share your models :D

---

For the plugin architecture we are going with a similar thinking. Plugins should provide a function that lets the user convert a fitted estimator to a fitted numpy based estimator. Similarly, if you fit with plugin X, then plugin X will be used to predict. Plugins can accept input with any type if they want to, so a GPU plugin can accept numpy arrays. So you can fit with cupy arrays on your GPU and then call `predict()` with a numpy array (if your plugin supports this). What is still a bit under discussion is whether you need to explicitly activate the plugin during `predict` or if it is implicitly activated because it was used for fitting.

Labels: API, RFC, Array API