Talip7/scikit-learn-issues-multilabel huggingface dataset:
Add Diebold Mariano test for distinguishing forecasts

### Describe the workflow you want to enable

I would like to be able to compare whether one forecast is statistically better than another.

### Describe your proposed solution

Under certain conditions, the *Diebold-Mariano* test achieves this. There's an example in Python [here](https://github.com/johntwk/Diebold-Mariano-Test).

### Describe alternatives you've considered, if relevant

I'm not sure there are alternatives to this.

### Additional context

In time series forecasting, we often want to know which forecast performs better. This test puts the difference in performance on a firm statistical footing.

COMMENT:
My personal take on this, is that this is unlikely to happen in scikit-learn so I am going to close this. More than happy to hear other opinions on this!

A few reasons to explain why I think this should be closed:
- full disclosure: I have never heard of it before, but it seems like this is more specific to forecasting time-series prediction which I would say is a bit outside of the main scikit-learn core focus
- the fact that you have asked pingouin to consider adding this feature and that the response from someone that seems to know this kind of test a lot better than me is that "The DM test is also very specific in a way" seems to imply that scikit-learn is not a good place for this see https://github.com/raphaelvallat/pingouin/issues/434#issuecomment-2295164845
- people tend to underestimate how much work is needed to add something new in scikit-learn (or similarly sized projects). In scikit-learn, there are currently 480+ issues opened with the label "New feature" and I would say most of them have a very small chance of ever being worked on, let alone being merged one day ...

I would encourage you to see if there is not a better package in the ecosystem which would make more sense to host this kind of functionality. Not sure whether that is useful for your use case but I found one project on PyPI https://pypi.org/project/dieboldmariano that maybe worth looking at. The repo you point to has some Python 2 specific code in their README which you know is generally not a great sign.

Labels: New Feature, Needs Triage
⚠️ CI failed on Wheel builder (last failure: Aug 17, 2024) ⚠️

**CI failed on [Wheel builder](https://github.com/scikit-learn/scikit-learn/actions/runs/10429290896)** (Aug 17, 2024)

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://github.com/scikit-learn/scikit-learn/actions/runs/10437578688) on Aug 18, 2024

Labels: Needs Triage
Arguments in train_test_split not being recognised.

### Describe the bug

When using the train_test_split function, arguments such as "test_size" and "random_state" are not being recognized, generating an unexpected keyword argument TypeError. 

### Steps/Code to Reproduce

```
 x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.2, random_state = 42) 
```
with x and y being PyTorch tensors

### Expected Results

The split occurs without an error, that is, the arguments are correctly recognised.

### Actual Results

```
... in train_test_split
    x_train, x_test, y_train, y_test = train_test_split(x, 
                                       ^^^^^^^^^^^^^^^^^^^
TypeError: train_test_split() got an unexpected keyword argument 'test_size'
```

### Versions

```shell
System:
    python: 3.12.4 | packaged by Anaconda, Inc. | (main, Jun 18 2024, 10:07:17) [Clang 14.0.6 ]
executable: /Users/name/miniconda3/bin/python3
   machine: macOS-12.5.1-arm64-arm-64bit

Python dependencies:
      sklearn: 1.5.1
          pip: 24.2
   setuptools: 69.5.1
        numpy: 1.26.4
        scipy: 1.14.0
       Cython: None
       pandas: 2.2.2
   matplotlib: 3.9.1
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 8
         prefix: libomp
       filepath: /Users/name/miniconda3/lib/python3.12/site-packages/sklearn/.dylibs/libomp.dylib
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libopenblas
       filepath: /Users/name/miniconda3/lib/libopenblasp-r0.3.21.dylib
        version: 0.3.21
threading_layer: pthreads
   architecture: armv8

       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libscipy_openblas
       filepath: /Users/name/miniconda3/lib/python3.12/site-packages/scipy/.dylibs/libscipy_openblas.dylib
        version: 0.3.27.dev
threading_layer: pthreads
   architecture: neoversen1
```

COMMENT:
I can't reproduce with scikit learn 1.5.1. Would you be able to post a [minimal reproducer](https://scikit-learn.org/dev/developers/minimal_reproducer.html), with the import and x and y defined?

(And can you post the whole error message)?

Labels: Bug, Needs Triage
Arguments in train_test_split not being recognised.

### Describe the bug

When using the train_test_split function, arguments such as "test_size" and "random_state" are not being recognized, generating an unexpected keyword argument TypeError. 

### Steps/Code to Reproduce

```
 x_train, x_test, y_train, y_test = train_test_split(x, y, test_size = 0.2, random_state = 42) 
```
with x and y being PyTorch tensors

### Expected Results

The split occurs without an error, that is, the arguments are correctly recognised.

### Actual Results

```
... in train_test_split
    x_train, x_test, y_train, y_test = train_test_split(x, 
                                       ^^^^^^^^^^^^^^^^^^^
TypeError: train_test_split() got an unexpected keyword argument 'test_size'
```

### Versions

```shell
System:
    python: 3.12.4 | packaged by Anaconda, Inc. | (main, Jun 18 2024, 10:07:17) [Clang 14.0.6 ]
executable: /Users/name/miniconda3/bin/python3
   machine: macOS-12.5.1-arm64-arm-64bit

Python dependencies:
      sklearn: 1.5.1
          pip: 24.2
   setuptools: 69.5.1
        numpy: 1.26.4
        scipy: 1.14.0
       Cython: None
       pandas: 2.2.2
   matplotlib: 3.9.1
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: openmp
   internal_api: openmp
    num_threads: 8
         prefix: libomp
       filepath: /Users/name/miniconda3/lib/python3.12/site-packages/sklearn/.dylibs/libomp.dylib
        version: None

       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libopenblas
       filepath: /Users/name/miniconda3/lib/libopenblasp-r0.3.21.dylib
        version: 0.3.21
threading_layer: pthreads
   architecture: armv8

       user_api: blas
   internal_api: openblas
    num_threads: 8
         prefix: libscipy_openblas
       filepath: /Users/name/miniconda3/lib/python3.12/site-packages/scipy/.dylibs/libscipy_openblas.dylib
        version: 0.3.27.dev
threading_layer: pthreads
   architecture: neoversen1
```

COMMENT:
I found an error in the script, which was causing this TypeError. I'm closing this issue, as the functioning is correct, just was a mistake by my side ;). Thanks by the help, anyway!

Labels: Bug, Needs Triage
Array API backends support for MLX

It would be great to get the scikit-learn Array API back-end to be compatible with MLX (which is mostly conformant with the array API). 

Here is an example which currently does not work for a few reasons:

```python
from sklearn.datasets import make_classification
from sklearn import config_context
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
import mlx.core as mx

X_np, y_np = make_classification(random_state=0)
X_mx = mx.array(X_np)
y_mx = mx.array(y_np)

with config_context(array_api_dispatch=True):
    lda = LinearDiscriminantAnalysis()
    X_trans = lda.fit_transform(X_mx, y_mx)

print(type(X_trans))
```

The reasons it does not work:

- MLX does not have a `float64` data type (similar to PyTorch MPS backend). It's a bit hacky to set `mx.float64 = mx.float32` so maybe good to handle this in the scikit or in a compatibility layer.

- MLX does not support operations with data-dependent output shapes, e.g. [`unique_values`](https://data-apis.org/array-api/2022.12/API_specification/generated/array_api.unique_values.html). Since these are optional in the array API should we attempt to avoid using them in scikit to get maximal compatibility with other frameworks?

- There are still a couple functions missing in MLX like `mx.asarray` and `mx.isdtype` (those are pretty easy for us to add)

Relevant discussion in MLX https://github.com/ml-explore/mlx/pull/1289

CC @betatim

COMMENT:
Does MLX plan to support array API? I don't think it'd be a good idea for us to support non-array-API-compatible libraries, since it'll ad quite a bit of maintenance burden and too many parallel code execution paths.

Labels: New Feature, Needs Investigation, Array API
Array API backends support for MLX

It would be great to get the scikit-learn Array API back-end to be compatible with MLX (which is mostly conformant with the array API). 

Here is an example which currently does not work for a few reasons:

```python
from sklearn.datasets import make_classification
from sklearn import config_context
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
import mlx.core as mx

X_np, y_np = make_classification(random_state=0)
X_mx = mx.array(X_np)
y_mx = mx.array(y_np)

with config_context(array_api_dispatch=True):
    lda = LinearDiscriminantAnalysis()
    X_trans = lda.fit_transform(X_mx, y_mx)

print(type(X_trans))
```

The reasons it does not work:

- MLX does not have a `float64` data type (similar to PyTorch MPS backend). It's a bit hacky to set `mx.float64 = mx.float32` so maybe good to handle this in the scikit or in a compatibility layer.

- MLX does not support operations with data-dependent output shapes, e.g. [`unique_values`](https://data-apis.org/array-api/2022.12/API_specification/generated/array_api.unique_values.html). Since these are optional in the array API should we attempt to avoid using them in scikit to get maximal compatibility with other frameworks?

- There are still a couple functions missing in MLX like `mx.asarray` and `mx.isdtype` (those are pretty easy for us to add)

Relevant discussion in MLX https://github.com/ml-explore/mlx/pull/1289

CC @betatim

COMMENT:
cc @ogrisel

Labels: New Feature, Needs Investigation, Array API
Array API backends support for MLX

It would be great to get the scikit-learn Array API back-end to be compatible with MLX (which is mostly conformant with the array API). 

Here is an example which currently does not work for a few reasons:

```python
from sklearn.datasets import make_classification
from sklearn import config_context
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
import mlx.core as mx

X_np, y_np = make_classification(random_state=0)
X_mx = mx.array(X_np)
y_mx = mx.array(y_np)

with config_context(array_api_dispatch=True):
    lda = LinearDiscriminantAnalysis()
    X_trans = lda.fit_transform(X_mx, y_mx)

print(type(X_trans))
```

The reasons it does not work:

- MLX does not have a `float64` data type (similar to PyTorch MPS backend). It's a bit hacky to set `mx.float64 = mx.float32` so maybe good to handle this in the scikit or in a compatibility layer.

- MLX does not support operations with data-dependent output shapes, e.g. [`unique_values`](https://data-apis.org/array-api/2022.12/API_specification/generated/array_api.unique_values.html). Since these are optional in the array API should we attempt to avoid using them in scikit to get maximal compatibility with other frameworks?

- There are still a couple functions missing in MLX like `mx.asarray` and `mx.isdtype` (those are pretty easy for us to add)

Relevant discussion in MLX https://github.com/ml-explore/mlx/pull/1289

CC @betatim

COMMENT:
Indeed, MLX plans to support the array API with two exceptions:

1. float64
2. Ops with data-dependent output shapes

I think you'll find those exceptions will not be uncommon (e.g. PyTorch MPS doesn't have 1, Jax doesn't have 2). So basically the point of this issue is to discuss how to handle Array API support in scikit-learn for frameworks which don't support 1 and 2.

Labels: New Feature, Needs Investigation, Array API
Array API backends support for MLX

It would be great to get the scikit-learn Array API back-end to be compatible with MLX (which is mostly conformant with the array API). 

Here is an example which currently does not work for a few reasons:

```python
from sklearn.datasets import make_classification
from sklearn import config_context
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
import mlx.core as mx

X_np, y_np = make_classification(random_state=0)
X_mx = mx.array(X_np)
y_mx = mx.array(y_np)

with config_context(array_api_dispatch=True):
    lda = LinearDiscriminantAnalysis()
    X_trans = lda.fit_transform(X_mx, y_mx)

print(type(X_trans))
```

The reasons it does not work:

- MLX does not have a `float64` data type (similar to PyTorch MPS backend). It's a bit hacky to set `mx.float64 = mx.float32` so maybe good to handle this in the scikit or in a compatibility layer.

- MLX does not support operations with data-dependent output shapes, e.g. [`unique_values`](https://data-apis.org/array-api/2022.12/API_specification/generated/array_api.unique_values.html). Since these are optional in the array API should we attempt to avoid using them in scikit to get maximal compatibility with other frameworks?

- There are still a couple functions missing in MLX like `mx.asarray` and `mx.isdtype` (those are pretty easy for us to add)

Relevant discussion in MLX https://github.com/ml-explore/mlx/pull/1289

CC @betatim

COMMENT:
Thanks for creating the issue and summarising the thread that lead to it!

I think we can address the "no `float64`" issue in the same way we addressed it for pytorch on MPS. Though this might be the moment where we should think about doing it "properly" (via the inspection API?). Instead of special casing libraries.

On the second point, I have no thoughts other than "hmm, interesting. Need to think about that". I don't think anyone has tried using jax with scikit-learn (yet). Ideally we can find one solution for all libraries.

Overall our experience has been that there are small differences/things that need to be dealt with, so right now we prefer to explicitly list the libraries that are supported. Yes, a bit weird maybe given the goal of the array API, but :-/ This means it is great to see someone come and lobby for mlx. It is a reason to think about the things that need to be done to support it, as well as things we might be doing right now that maybe we shouldn't.

Labels: New Feature, Needs Investigation, Array API
Array API backends support for MLX

It would be great to get the scikit-learn Array API back-end to be compatible with MLX (which is mostly conformant with the array API). 

Here is an example which currently does not work for a few reasons:

```python
from sklearn.datasets import make_classification
from sklearn import config_context
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
import mlx.core as mx

X_np, y_np = make_classification(random_state=0)
X_mx = mx.array(X_np)
y_mx = mx.array(y_np)

with config_context(array_api_dispatch=True):
    lda = LinearDiscriminantAnalysis()
    X_trans = lda.fit_transform(X_mx, y_mx)

print(type(X_trans))
```

The reasons it does not work:

- MLX does not have a `float64` data type (similar to PyTorch MPS backend). It's a bit hacky to set `mx.float64 = mx.float32` so maybe good to handle this in the scikit or in a compatibility layer.

- MLX does not support operations with data-dependent output shapes, e.g. [`unique_values`](https://data-apis.org/array-api/2022.12/API_specification/generated/array_api.unique_values.html). Since these are optional in the array API should we attempt to avoid using them in scikit to get maximal compatibility with other frameworks?

- There are still a couple functions missing in MLX like `mx.asarray` and `mx.isdtype` (those are pretty easy for us to add)

Relevant discussion in MLX https://github.com/ml-explore/mlx/pull/1289

CC @betatim

COMMENT:
> I think we can address the "no float64" issue in the same way we addressed it for pytorch on MPS. Though this might be the moment where we should think about doing it "properly" (via the inspection API?). Instead of special casing libraries.

It's not exactly the same though. `torch.float64` (or `array_api_compat.torch.float64` as actually used in scikit-learn) does exist as a public attribute on the namespace module but `torch.float64` operations are not supported on tensors that are allocated on a MPS device (but they are on CUDA and CPU devices). For mlx, the `mlx.core.float64` symbol does not even exist.

At this time we do have library specific code to handle the torch MPS constraint:

https://github.com/scikit-learn/scikit-learn/blob/d9deffef00e0a2001e0c93fc99124028508f4557/sklearn/utils/_array_api.py#L650-L659

As Tim said we could extend that code to support mlx if that was the only problem.

> Ops with data-dependent output shapes

At this time we have several operations in scikit-learn that require data-dependent output shapes. We have the same problem with Dask and JAX support. More details in the following stalled experimental PRs:

- #28588
- #29647

I cannot suggest an easy forward at this point. More investigation/experiments are required to propose solutions to unlock this class of problems, ideally in a library agnostic fashion.

EDIT: actually the JAX inplace assignment limitation is different. It's possible that the `xp.unique_values` problem could be worked around more easily in limited number of places in the scikit-learn code base.

Labels: New Feature, Needs Investigation, Array API
Array API backends support for MLX

It would be great to get the scikit-learn Array API back-end to be compatible with MLX (which is mostly conformant with the array API). 

Here is an example which currently does not work for a few reasons:

```python
from sklearn.datasets import make_classification
from sklearn import config_context
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
import mlx.core as mx

X_np, y_np = make_classification(random_state=0)
X_mx = mx.array(X_np)
y_mx = mx.array(y_np)

with config_context(array_api_dispatch=True):
    lda = LinearDiscriminantAnalysis()
    X_trans = lda.fit_transform(X_mx, y_mx)

print(type(X_trans))
```

The reasons it does not work:

- MLX does not have a `float64` data type (similar to PyTorch MPS backend). It's a bit hacky to set `mx.float64 = mx.float32` so maybe good to handle this in the scikit or in a compatibility layer.

- MLX does not support operations with data-dependent output shapes, e.g. [`unique_values`](https://data-apis.org/array-api/2022.12/API_specification/generated/array_api.unique_values.html). Since these are optional in the array API should we attempt to avoid using them in scikit to get maximal compatibility with other frameworks?

- There are still a couple functions missing in MLX like `mx.asarray` and `mx.isdtype` (those are pretty easy for us to add)

Relevant discussion in MLX https://github.com/ml-explore/mlx/pull/1289

CC @betatim

COMMENT:
> maybe good to handle this in the scikit or in a compatibility layer.

It would be useful to take a look at wrapping MLX in array-api-compat (see e.g. https://github.com/data-apis/array-api-compat/pull/76 for Dask). Then you can make changes like `float64 = mlx.float32` and see what else fails the array API test suite.

Labels: New Feature, Needs Investigation, Array API
Array API backends support for MLX

It would be great to get the scikit-learn Array API back-end to be compatible with MLX (which is mostly conformant with the array API). 

Here is an example which currently does not work for a few reasons:

```python
from sklearn.datasets import make_classification
from sklearn import config_context
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
import mlx.core as mx

X_np, y_np = make_classification(random_state=0)
X_mx = mx.array(X_np)
y_mx = mx.array(y_np)

with config_context(array_api_dispatch=True):
    lda = LinearDiscriminantAnalysis()
    X_trans = lda.fit_transform(X_mx, y_mx)

print(type(X_trans))
```

The reasons it does not work:

- MLX does not have a `float64` data type (similar to PyTorch MPS backend). It's a bit hacky to set `mx.float64 = mx.float32` so maybe good to handle this in the scikit or in a compatibility layer.

- MLX does not support operations with data-dependent output shapes, e.g. [`unique_values`](https://data-apis.org/array-api/2022.12/API_specification/generated/array_api.unique_values.html). Since these are optional in the array API should we attempt to avoid using them in scikit to get maximal compatibility with other frameworks?

- There are still a couple functions missing in MLX like `mx.asarray` and `mx.isdtype` (those are pretty easy for us to add)

Relevant discussion in MLX https://github.com/ml-explore/mlx/pull/1289

CC @betatim

COMMENT:
> Exposing a fake xp.float64 symbol in a compat layer might work but feels weird. It might be a pragmatic first step into exploring what else would break in practice once this technical detail is sidestepped.

👍.

@j-emberton are you still interested in working on a compat layer for MLX?

Labels: New Feature, Needs Investigation, Array API
Array API backends support for MLX

It would be great to get the scikit-learn Array API back-end to be compatible with MLX (which is mostly conformant with the array API). 

Here is an example which currently does not work for a few reasons:

```python
from sklearn.datasets import make_classification
from sklearn import config_context
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
import mlx.core as mx

X_np, y_np = make_classification(random_state=0)
X_mx = mx.array(X_np)
y_mx = mx.array(y_np)

with config_context(array_api_dispatch=True):
    lda = LinearDiscriminantAnalysis()
    X_trans = lda.fit_transform(X_mx, y_mx)

print(type(X_trans))
```

The reasons it does not work:

- MLX does not have a `float64` data type (similar to PyTorch MPS backend). It's a bit hacky to set `mx.float64 = mx.float32` so maybe good to handle this in the scikit or in a compatibility layer.

- MLX does not support operations with data-dependent output shapes, e.g. [`unique_values`](https://data-apis.org/array-api/2022.12/API_specification/generated/array_api.unique_values.html). Since these are optional in the array API should we attempt to avoid using them in scikit to get maximal compatibility with other frameworks?

- There are still a couple functions missing in MLX like `mx.asarray` and `mx.isdtype` (those are pretty easy for us to add)

Relevant discussion in MLX https://github.com/ml-explore/mlx/pull/1289

CC @betatim

COMMENT:
That would be great! And functions which you think MLX should have that are part of the array API but missing we can add. Just let us know.

Labels: New Feature, Needs Investigation, Array API