Talip7/scikit-learn-issues-multilabel huggingface dataset:
Automatically move `y_true` to the same device and namespace as `y_pred` for metrics

This is closely linked to #28668 but separate enough to warrant it's own issue (https://github.com/scikit-learn/scikit-learn/issues/28668#issuecomment-2814771519). This is mostly a summary of discussions so far. If we are happy with a decision, we can move to updating the documentation.

---

For classification metrics to support array API, there is a problem in the case where `y_pred` is not in the same namespace/device as `y_true`.

`y_pred` is likely to be the output of `predict_proba` or `decision_function` and would be in the same namespace/device as `X` (if we decide in #28668 that "everything should follow X").
`y_true` could be an integer array or a numpy array or pandas series (this is pertinent as `y_true` may be string labels)

Motivating use case:

Using e.g., `GridSearchCV` or `cross_validate` with a pipeline that moves `X` to GPU.
Consider a pipeline like below (copied from https://github.com/scikit-learn/scikit-learn/issues/28668#issuecomment-2154958666): 

```python
pipeline = make_pipeline(
   SomeDataFrameAwareFeatureExtractor(),
   MoveFeaturesToPyTorch(device="cuda"),
   SomeArrayAPICapableClassifier(),
)
```

Pipelines do not ever touch `y` so we are not able to alter `y` within the pipeline.
We would need to pass a metric to `GridSearchCV` or `cross_validate`, which would be passed `y_true` and `y_pred` on different namespace / devices.

Thus the motivation to automatically move `y_true` to the same namespace / device as `y_pred`, in metrics functions.

(Note another example is discussed in https://github.com/scikit-learn/scikit-learn/pull/30439#issuecomment-2531072292)

As it is more likely that `y_pred` is on GPU, `y_true` follow `y_pred` was slightly preferred over `y_pred` follows `y_true`. Computation wise, CPU vs GPU is probably similar for metrics like log-loss, but for metrics that require sorting (e.g., ROC AUC) GPU may be faster? (see https://github.com/scikit-learn/scikit-learn/pull/30439#issuecomment-2532238196 for more discussion on this point)


Question for my own clarification, the main motivation is for usability, so the user does not have to manually convert `y_true` ? Would a helper function to help the user convert `y_true` to the correct namespace/device be interesting?


cc @ogrisel @betatim

COMMENT:
+1!

Labels: API, Array API
Automatically move `y_true` to the same device and namespace as `y_pred` for metrics

This is closely linked to #28668 but separate enough to warrant it's own issue (https://github.com/scikit-learn/scikit-learn/issues/28668#issuecomment-2814771519). This is mostly a summary of discussions so far. If we are happy with a decision, we can move to updating the documentation.

---

For classification metrics to support array API, there is a problem in the case where `y_pred` is not in the same namespace/device as `y_true`.

`y_pred` is likely to be the output of `predict_proba` or `decision_function` and would be in the same namespace/device as `X` (if we decide in #28668 that "everything should follow X").
`y_true` could be an integer array or a numpy array or pandas series (this is pertinent as `y_true` may be string labels)

Motivating use case:

Using e.g., `GridSearchCV` or `cross_validate` with a pipeline that moves `X` to GPU.
Consider a pipeline like below (copied from https://github.com/scikit-learn/scikit-learn/issues/28668#issuecomment-2154958666): 

```python
pipeline = make_pipeline(
   SomeDataFrameAwareFeatureExtractor(),
   MoveFeaturesToPyTorch(device="cuda"),
   SomeArrayAPICapableClassifier(),
)
```

Pipelines do not ever touch `y` so we are not able to alter `y` within the pipeline.
We would need to pass a metric to `GridSearchCV` or `cross_validate`, which would be passed `y_true` and `y_pred` on different namespace / devices.

Thus the motivation to automatically move `y_true` to the same namespace / device as `y_pred`, in metrics functions.

(Note another example is discussed in https://github.com/scikit-learn/scikit-learn/pull/30439#issuecomment-2531072292)

As it is more likely that `y_pred` is on GPU, `y_true` follow `y_pred` was slightly preferred over `y_pred` follows `y_true`. Computation wise, CPU vs GPU is probably similar for metrics like log-loss, but for metrics that require sorting (e.g., ROC AUC) GPU may be faster? (see https://github.com/scikit-learn/scikit-learn/pull/30439#issuecomment-2532238196 for more discussion on this point)


Question for my own clarification, the main motivation is for usability, so the user does not have to manually convert `y_true` ? Would a helper function to help the user convert `y_true` to the correct namespace/device be interesting?


cc @ogrisel @betatim

COMMENT:
I think this sounds fine.

Labels: API, Array API
⚠️ CI failed on Wheel builder (last failure: May 05, 2025) ⚠️

**CI is still failing on [Wheel builder](https://github.com/scikit-learn/scikit-learn/actions/runs/14828681637)** (May 05, 2025)

COMMENT:
At the time of writing, #31263 was merged 17 hours ago and this failed 3 hours ago, so the fix was not enough. There is something different between our Azure Pipelines-based `[free-threeded]` CI config and the nightly wheels config that should explain this.

The nightly wheel builder uses developer versions of dependencies. I tried to investigate a bit in this comment of the preview automated issue but could not spot the cause by comparing the logs of running and failing nightly CI runs: https://github.com/scikit-learn/scikit-learn/issues/31257#issuecomment-2835076702

Labels: Build / CI
⚠️ CI failed on Wheel builder (last failure: May 05, 2025) ⚠️

**CI is still failing on [Wheel builder](https://github.com/scikit-learn/scikit-learn/actions/runs/14828681637)** (May 05, 2025)

COMMENT:
Direct links to:

- last successful nightly run (for manylinux):

https://github.com/scikit-learn/scikit-learn/actions/runs/14677414091/job/41195707673

- first failing nightly run (for manylinux):

https://github.com/scikit-learn/scikit-learn/actions/runs/14688110971/job/41219558777

- History of all the nightly wheels runs:

https://github.com/scikit-learn/scikit-learn/actions/workflows/wheels.yml?query=branch%3Amain+event%3Aschedule

Labels: Build / CI
⚠️ CI failed on Wheel builder (last failure: May 05, 2025) ⚠️

**CI is still failing on [Wheel builder](https://github.com/scikit-learn/scikit-learn/actions/runs/14828681637)** (May 05, 2025)

COMMENT:
This seems to be related to the developer version of Cython. Draft PR to investigate has been opened in #31300.

This can now be reproduced without free-threaded CPython so it's not related to free-threading.

Labels: Build / CI
Change the default data directory

### Describe the workflow you want to enable

It's not a good practice to put files directly into the home directory.

### Describe your proposed solution

A more common way is to put them into the standard cache directories recommended by operating systems:

| OS | Path |
| -- | ---- |
| Linux | `$XDG_CACHE_HOME` (if the env var presents) or `~/.cache` |
| macOS | `~/Library/Caches` |
| Windows | `%LOCALAPPDATA%` (`~/AppData/Local`) |

### Describe alternatives you've considered, if relevant

Put into `~/.cache/scikit-learn` for all operating systems. Though not being standard, it's still better than the home dir.

### Additional context

https://scikit-learn.org/stable/modules/generated/sklearn.datasets.get_data_home.html

COMMENT:
I agree this is a good idea. Following system conventions help tools and people looking for suggestions for cleanups when a disk is nearly full.

+1 for a PR on my side.

Labels: New Feature
Change the default data directory

### Describe the workflow you want to enable

It's not a good practice to put files directly into the home directory.

### Describe your proposed solution

A more common way is to put them into the standard cache directories recommended by operating systems:

| OS | Path |
| -- | ---- |
| Linux | `$XDG_CACHE_HOME` (if the env var presents) or `~/.cache` |
| macOS | `~/Library/Caches` |
| Windows | `%LOCALAPPDATA%` (`~/AppData/Local`) |

### Describe alternatives you've considered, if relevant

Put into `~/.cache/scikit-learn` for all operating systems. Though not being standard, it's still better than the home dir.

### Additional context

https://scikit-learn.org/stable/modules/generated/sklearn.datasets.get_data_home.html

COMMENT:
I made a pr for this.

The cache directory is named scikit_learn instead of scikit_learn_data as it better aligns with OS cache naming conventions.

Labels: New Feature
Change the default data directory

### Describe the workflow you want to enable

It's not a good practice to put files directly into the home directory.

### Describe your proposed solution

A more common way is to put them into the standard cache directories recommended by operating systems:

| OS | Path |
| -- | ---- |
| Linux | `$XDG_CACHE_HOME` (if the env var presents) or `~/.cache` |
| macOS | `~/Library/Caches` |
| Windows | `%LOCALAPPDATA%` (`~/AppData/Local`) |

### Describe alternatives you've considered, if relevant

Put into `~/.cache/scikit-learn` for all operating systems. Though not being standard, it's still better than the home dir.

### Additional context

https://scikit-learn.org/stable/modules/generated/sklearn.datasets.get_data_home.html

COMMENT:
Hey @balthild sent a pr

Labels: New Feature
⚠️ CI failed on Wheel builder (last failure: Apr 28, 2025) ⚠️

**CI is still failing on [Wheel builder](https://github.com/scikit-learn/scikit-learn/actions/runs/14699848568)** (Apr 28, 2025)

COMMENT:
The nightly CI has discovered a Cython-related problem on all the free-threading builds:

```python-traceback
   _______________ ERROR collecting utils/tests/test_cython_blas.py _______________
  ImportError while importing test module '/tmp/tmp.O8xqw5JN27/venv/lib/python3.13t/site-packages/sklearn/utils/tests/test_cython_blas.py'.
  Hint: make sure your test modules/packages have valid Python names.
  Traceback:
  /opt/_internal/cpython-3.13.3-nogil/lib/python3.13t/importlib/__init__.py:88: in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
  ../venv/lib/python3.13t/site-packages/sklearn/utils/tests/test_cython_blas.py:4: in <module>
      from sklearn.utils._cython_blas import (
  E   ImportError: cannot import name 'ColMajor' from 'sklearn.utils._cython_blas' (/tmp/tmp.O8xqw5JN27/venv/lib/python3.13t/site-packages/sklearn/utils/_cython_blas.cpython-313t-x86_64-linux-gnu.so)
```

Labels: Bug, free-threading
⚠️ CI failed on Wheel builder (last failure: Apr 28, 2025) ⚠️

**CI is still failing on [Wheel builder](https://github.com/scikit-learn/scikit-learn/actions/runs/14699848568)** (Apr 28, 2025)

COMMENT:
`ColMajor` comes from:

https://github.com/scikit-learn/scikit-learn/blob/7131d9488dfb8edd6ae042caca57dd76523f395b/sklearn/utils/_cython_blas.pxd#L4-L6

Labels: Bug, free-threading
⚠️ CI failed on Wheel builder (last failure: Apr 28, 2025) ⚠️

**CI is still failing on [Wheel builder](https://github.com/scikit-learn/scikit-learn/actions/runs/14699848568)** (Apr 28, 2025)

COMMENT:
I am not sure exactly what caused this build to start failing 2 days ago. Here is the history of the runs:

https://github.com/scikit-learn/scikit-learn/actions/workflows/wheels.yml?query=branch%3Amain+event%3Aschedule

Labels: Bug, free-threading
⚠️ CI failed on Wheel builder (last failure: Apr 28, 2025) ⚠️

**CI is still failing on [Wheel builder](https://github.com/scikit-learn/scikit-learn/actions/runs/14699848568)** (Apr 28, 2025)

COMMENT:
In the last successful run and the rist failing run, the Python versions were both:

```
3.13.2 experimental free-threading build (main, Mar 22 2025, 16:18:34) [GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]
```

and the wheel was built with:

```
     Successfully installed Cython-3.1.0b1 meson-1.8.0rc2 meson-python-0.17.1 numpy-2.3.0.dev0 packaging-25.0 pyproject-metadata-0.9.1 scipy-1.16.0.dev0
```

in both cases. Maybe it's a side effect of a change in SciPy?

Anyway the fix in #31263 seems legit and I wonder why Cython does not complain at build time.

Labels: Bug, free-threading
⚠️ CI failed on Linux_Runs.pylatest_conda_forge_mkl (last failure: Apr 26, 2025) ⚠️

**CI failed on [Linux_Runs.pylatest_conda_forge_mkl](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=75987&view=logs&j=dde5042c-7464-5d47-9507-31bdd2ee0a3a)** (Apr 26, 2025)
- test_precomputed_nearest_neighbors_filtering[60]

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=76093&view=logs&j=dde5042c-7464-5d47-9507-31bdd2ee0a3a) on Apr 30, 2025

Labels: module:test-suite
⚠️ CI failed on Linux_Runs.pylatest_conda_forge_mkl (last failure: Apr 26, 2025) ⚠️

**CI failed on [Linux_Runs.pylatest_conda_forge_mkl](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=75987&view=logs&j=dde5042c-7464-5d47-9507-31bdd2ee0a3a)** (Apr 26, 2025)
- test_precomputed_nearest_neighbors_filtering[60]

COMMENT:
I ran locally (on arm64 macOS) this test will all admissible random seeds and it passes for me. So this is likely a case of platform/rng seed cross-sensitivity.

Labels: module:test-suite
⚠️ CI failed on Linux_Runs.pylatest_conda_forge_mkl (last failure: Apr 26, 2025) ⚠️

**CI failed on [Linux_Runs.pylatest_conda_forge_mkl](https://dev.azure.com/scikit-learn/scikit-learn/_build/results?buildId=75987&view=logs&j=dde5042c-7464-5d47-9507-31bdd2ee0a3a)** (Apr 26, 2025)
- test_precomputed_nearest_neighbors_filtering[60]

COMMENT:
Fixed by https://github.com/scikit-learn/scikit-learn/pull/31262

Labels: module:test-suite
Faster Eigen Decomposition for Isomap & KernelPCA

(disclaimer: this issue and associated PR are part of a student project supervised by @smarie )

### Summary

Eigendecomposition is slow when number of samples is large. This impacts decomposition models such as KernelPCA and Isomap. A "randomized" eigendecomposition method (from [Halko et al](https://arxiv.org/abs/0909.4061)) [has been introduced for KernelPCA](https://scikit-learn.org/stable/modules/decomposition.html#choice-of-solver-for-kernel-pca) leveraging Halko's algorithm 4.3 for randomized SVD decomposition (also used in [PCA](https://scikit-learn.org/stable/modules/decomposition.html#pca-using-randomized-svd)).

Unfortunately, the current approach is only valid for decomposition of PSD matrices - which suits well for KernelPCA but can not be true in the context of Isomap. Therefore Isomap has not accelerated implementation as of today.

We propose to introduce an additional approximate eigendecomposition method based on algorithm 5.3 from the same paper.
This method should offer a faster alternative to existing solvers (arpack, dense, etc.) while maintaining accuracy, and as opposed to randomized svd, is suitable to find eigenvalues for non-PSD matrices.

### Describe your proposed solution

- Implement `_randomized_eigsh(selection='value')`, that is left as [NotImplemented](https://github.com/scikit-learn/scikit-learn/pull/12069) today.
- Integrate it as an alternate solver in `Isomap` and in `KernelPCA`.
- Add tests comparing performance with existing solvers.
- Provide benchmarks to evaluate speedup and accuracy.


### Motivation

- Improves scalability for large datasets.
- Reduces computation time for eigen decomposition-based methods.

Note: this solution could be used to accelerate all models relying on eigenvalue decomposition, including possibly https://github.com/scikit-learn/scikit-learn/pull/22330

COMMENT:
I am not familiar (anymore) with the details of `KernelPCA`, `Isomap`, `_randomized_eighs` and algorithm 5.3 in general, but what you propose sounds reasonable.

Looking forward to reviewing the PR (I assume #31247 is the PR by your student) and once the CI is green and once it includes some benchmark results.

Labels: New Feature
Faster Eigen Decomposition for Isomap & KernelPCA

(disclaimer: this issue and associated PR are part of a student project supervised by @smarie )

### Summary

Eigendecomposition is slow when number of samples is large. This impacts decomposition models such as KernelPCA and Isomap. A "randomized" eigendecomposition method (from [Halko et al](https://arxiv.org/abs/0909.4061)) [has been introduced for KernelPCA](https://scikit-learn.org/stable/modules/decomposition.html#choice-of-solver-for-kernel-pca) leveraging Halko's algorithm 4.3 for randomized SVD decomposition (also used in [PCA](https://scikit-learn.org/stable/modules/decomposition.html#pca-using-randomized-svd)).

Unfortunately, the current approach is only valid for decomposition of PSD matrices - which suits well for KernelPCA but can not be true in the context of Isomap. Therefore Isomap has not accelerated implementation as of today.

We propose to introduce an additional approximate eigendecomposition method based on algorithm 5.3 from the same paper.
This method should offer a faster alternative to existing solvers (arpack, dense, etc.) while maintaining accuracy, and as opposed to randomized svd, is suitable to find eigenvalues for non-PSD matrices.

### Describe your proposed solution

- Implement `_randomized_eigsh(selection='value')`, that is left as [NotImplemented](https://github.com/scikit-learn/scikit-learn/pull/12069) today.
- Integrate it as an alternate solver in `Isomap` and in `KernelPCA`.
- Add tests comparing performance with existing solvers.
- Provide benchmarks to evaluate speedup and accuracy.


### Motivation

- Improves scalability for large datasets.
- Reduces computation time for eigen decomposition-based methods.

Note: this solution could be used to accelerate all models relying on eigenvalue decomposition, including possibly https://github.com/scikit-learn/scikit-learn/pull/22330

COMMENT:
Thanks @ogrisel for the quick feedback !

Labels: New Feature
GradientBoostingClassifier does not have out-of-bag (OOB) score

### Describe the bug

Hi, the [documentation page](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html) for Gradient boosting Classifier says that there is an out-of-bag score that can be retrieved by the `oob_score_` attribute. However, this attribute doesn't seem to exist in the latest version.



### Steps/Code to Reproduce

Copy-and-paste code to reproduce this:

```python
from sklearn.ensemble import GradientBoostingClassifier
import numpy as np

Xs = np.random.randn(100, 10)
ys = np.random.randint(0, 2, 100)

gbc = GradientBoostingClassifier()
gbc.fit(Xs, ys)
gbc.oob_score_
```

### Expected Results

No error is thrown. OOB score should be a float

### Actual Results

```
$ conda create -n sklearn-env -c conda-forge scikit-learn
$ conda activate sklearn-env
(sklearn-env) $ python
Python 3.13.3 | packaged by conda-forge | (main, Apr 14 2025, 20:44:30) [Clang 18.1.8 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sklearn; sklearn.__version__
'1.6.1'
>>> from sklearn.ensemble import GradientBoostingClassifier
... import numpy as np
...
... Xs = np.random.randn(100, 10)
... ys = np.random.randint(0, 2, 100)
...
... gbc = GradientBoostingClassifier()
... gbc.fit(Xs, ys)
... gbc.oob_score_
Traceback (most recent call last):
  File "<python-input-0>", line 9, in <module>
    gbc.oob_score_
AttributeError: 'GradientBoostingClassifier' object has no attribute 'oob_score_'
```

### Versions

```shell
>>> import sklearn; sklearn.show_versions()

System:
    python: 3.13.3 | packaged by conda-forge | (main, Apr 14 2025, 20:44:30) [Clang 18.1.8 ]
executable: /Users/longyuxi/miniforge3/envs/sklearn-env/bin/python
   machine: macOS-15.3.2-arm64-arm-64bit-Mach-O

Python dependencies:
      sklearn: 1.6.1
          pip: 25.0.1
   setuptools: 79.0.1
        numpy: 2.2.5
        scipy: 1.15.2
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.4.2
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 10
         prefix: libopenblas
       filepath: /Users/longyuxi/miniforge3/envs/sklearn-env/lib/libopenblas.0.dylib
        version: 0.3.29
threading_layer: openmp
   architecture: VORTEX

       user_api: openmp
   internal_api: openmp
    num_threads: 10
         prefix: libomp
       filepath: /Users/longyuxi/miniforge3/envs/sklearn-env/lib/libomp.dylib
        version: None
```

COMMENT:
I am so confused. I tried the following configurations on my M4 mac:

1. sklearn 1.6 installed with conda (above)
2. sklearn 1.3 installed with conda
3. sklearn 1.6 installed with pip

and all of them throw the same attribute not found error as above, even though this feature was supposed to be added in 1.3.

I also tried this on an intel-based machine and got the same error.

Labels: Bug, Needs Triage
GradientBoostingClassifier does not have out-of-bag (OOB) score

### Describe the bug

Hi, the [documentation page](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html) for Gradient boosting Classifier says that there is an out-of-bag score that can be retrieved by the `oob_score_` attribute. However, this attribute doesn't seem to exist in the latest version.



### Steps/Code to Reproduce

Copy-and-paste code to reproduce this:

```python
from sklearn.ensemble import GradientBoostingClassifier
import numpy as np

Xs = np.random.randn(100, 10)
ys = np.random.randint(0, 2, 100)

gbc = GradientBoostingClassifier()
gbc.fit(Xs, ys)
gbc.oob_score_
```

### Expected Results

No error is thrown. OOB score should be a float

### Actual Results

```
$ conda create -n sklearn-env -c conda-forge scikit-learn
$ conda activate sklearn-env
(sklearn-env) $ python
Python 3.13.3 | packaged by conda-forge | (main, Apr 14 2025, 20:44:30) [Clang 18.1.8 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import sklearn; sklearn.__version__
'1.6.1'
>>> from sklearn.ensemble import GradientBoostingClassifier
... import numpy as np
...
... Xs = np.random.randn(100, 10)
... ys = np.random.randint(0, 2, 100)
...
... gbc = GradientBoostingClassifier()
... gbc.fit(Xs, ys)
... gbc.oob_score_
Traceback (most recent call last):
  File "<python-input-0>", line 9, in <module>
    gbc.oob_score_
AttributeError: 'GradientBoostingClassifier' object has no attribute 'oob_score_'
```

### Versions

```shell
>>> import sklearn; sklearn.show_versions()

System:
    python: 3.13.3 | packaged by conda-forge | (main, Apr 14 2025, 20:44:30) [Clang 18.1.8 ]
executable: /Users/longyuxi/miniforge3/envs/sklearn-env/bin/python
   machine: macOS-15.3.2-arm64-arm-64bit-Mach-O

Python dependencies:
      sklearn: 1.6.1
          pip: 25.0.1
   setuptools: 79.0.1
        numpy: 2.2.5
        scipy: 1.15.2
       Cython: None
       pandas: None
   matplotlib: None
       joblib: 1.4.2
threadpoolctl: 3.6.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 10
         prefix: libopenblas
       filepath: /Users/longyuxi/miniforge3/envs/sklearn-env/lib/libopenblas.0.dylib
        version: 0.3.29
threading_layer: openmp
   architecture: VORTEX

       user_api: openmp
   internal_api: openmp
    num_threads: 10
         prefix: libomp
       filepath: /Users/longyuxi/miniforge3/envs/sklearn-env/lib/libomp.dylib
        version: None
```

COMMENT:
Okay, I found out that I need to specify the parameter for `subsample` to be something less than 1 to get the score.

Labels: Bug, Needs Triage