Talip7/scikit-learn-issues-multilabel huggingface dataset:
[BUG] `roc_auc_score` is wrong (edge case)

### Describe the bug

Hi,

It seems that on the edge case when there are equal scores, [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#roc-auc-score) actually makes a wrong comuptation:
```pycon
>>> roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
0.375
>>> # Expected 0.25 or 0.5, see below.
```
On would expect either `0.5` or `0.25` depending on whether or not the area is computed by interpolating to the convex hull (which makes sense with probabilistic mixtures for the ROC curve) or not.

I tried illustrating this example below (keep in mind that this behaviour is *speculated* by myself, I didn't go through any of `sklearn` code).

![roc_auc_bug](https://github.com/scikit-learn/scikit-learn/assets/114467748/4a902ca2-fc23-4341-b8b9-6234a9f40fd8)

Any feedback is welcome.
Cheers!

### Steps/Code to Reproduce

```pycon
from sklearn.metrics import roc_auc_score
roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
```

### Expected Results

`0.25` or `0.5`

### Actual Results

`0.375`

### Versions

```shell
1.5.0
```

COMMENT:
And the extreme case of this situation is the constant predictor that always predicts the majority class of the training set (only exact tied predictions). In this case we all agree that it's best represented by the "chance level" diagonal ROC curve. Using `kind="previous"` interpolation for this baseline classifiers would be misleading. We all agree that non-informative classifiers should get a ROC AUC value of 0.5.

Labels: Bug
[BUG] `roc_auc_score` is wrong (edge case)

### Describe the bug

Hi,

It seems that on the edge case when there are equal scores, [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#roc-auc-score) actually makes a wrong comuptation:
```pycon
>>> roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
0.375
>>> # Expected 0.25 or 0.5, see below.
```
On would expect either `0.5` or `0.25` depending on whether or not the area is computed by interpolating to the convex hull (which makes sense with probabilistic mixtures for the ROC curve) or not.

I tried illustrating this example below (keep in mind that this behaviour is *speculated* by myself, I didn't go through any of `sklearn` code).

![roc_auc_bug](https://github.com/scikit-learn/scikit-learn/assets/114467748/4a902ca2-fc23-4341-b8b9-6234a9f40fd8)

Any feedback is welcome.
Cheers!

### Steps/Code to Reproduce

```pycon
from sklearn.metrics import roc_auc_score
roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
```

### Expected Results

`0.25` or `0.5`

### Actual Results

`0.375`

### Versions

```shell
1.5.0
```

COMMENT:
Yes, the current implementation is similar to breaking ties stocastically **while** still conserving deterministic single-thresholded behaviour, which doesn't really make sense in my opinion. Especially if you consider scoring functions that always take integer values (could be a count of nearest neighbours for example, or anything else really).

> And the extreme case of this situation is the constant predictor that always predicts the majority class of the training set (only exact tied predictions). In this case we all agree that it's best represented by the "chance level" diagonal ROC curve. Using `kind="previous"` interpolation for this baseline classifiers would be misleading. We all agree that non-informative classifiers should get a ROC AUC value of 0.5.

Exactly, which is why in general, I think the correct definition of the ROC curve should be defined by the convex hull, which corresponds to linear interpolation on the Pareto front.

I'm not sure it's a heavy modification to make if you're okay with using scipy's [`ConvexHull`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.spatial.ConvexHull.html) functionality. This can even be done with elementary algebra since we're in a simplified case.

Labels: Bug
[BUG] `roc_auc_score` is wrong (edge case)

### Describe the bug

Hi,

It seems that on the edge case when there are equal scores, [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#roc-auc-score) actually makes a wrong comuptation:
```pycon
>>> roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
0.375
>>> # Expected 0.25 or 0.5, see below.
```
On would expect either `0.5` or `0.25` depending on whether or not the area is computed by interpolating to the convex hull (which makes sense with probabilistic mixtures for the ROC curve) or not.

I tried illustrating this example below (keep in mind that this behaviour is *speculated* by myself, I didn't go through any of `sklearn` code).

![roc_auc_bug](https://github.com/scikit-learn/scikit-learn/assets/114467748/4a902ca2-fc23-4341-b8b9-6234a9f40fd8)

Any feedback is welcome.
Cheers!

### Steps/Code to Reproduce

```pycon
from sklearn.metrics import roc_auc_score
roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
```

### Expected Results

`0.25` or `0.5`

### Actual Results

`0.375`

### Versions

```shell
1.5.0
```

COMMENT:
> Exactly, which is why in general, I think the correct definition of the ROC curve should be defined by the convex hull, which corresponds to linear interpolation on the Pareto front.

Alright but we cannot rename things that are so common in the literature. We can introduce `rocch_curve` and `rocch_auc` though, as new functions in scikit-learn, or add an extra kwarg to enable the Convex Hull version of ROC on the existing tools.

Labels: Bug
[BUG] `roc_auc_score` is wrong (edge case)

### Describe the bug

Hi,

It seems that on the edge case when there are equal scores, [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#roc-auc-score) actually makes a wrong comuptation:
```pycon
>>> roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
0.375
>>> # Expected 0.25 or 0.5, see below.
```
On would expect either `0.5` or `0.25` depending on whether or not the area is computed by interpolating to the convex hull (which makes sense with probabilistic mixtures for the ROC curve) or not.

I tried illustrating this example below (keep in mind that this behaviour is *speculated* by myself, I didn't go through any of `sklearn` code).

![roc_auc_bug](https://github.com/scikit-learn/scikit-learn/assets/114467748/4a902ca2-fc23-4341-b8b9-6234a9f40fd8)

Any feedback is welcome.
Cheers!

### Steps/Code to Reproduce

```pycon
from sklearn.metrics import roc_auc_score
roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
```

### Expected Results

`0.25` or `0.5`

### Actual Results

`0.375`

### Versions

```shell
1.5.0
```

COMMENT:
I agree we ccannot rename common grounds as we please but I think it is an important question here: what **is** the common ground? Can you find somewhere a proper definition of the ROC curve? I have not been able to. People agree on the points `(FPR, TPR)`, but what to do with those: `previous` interpolation, `linear` interpolation, `convex hull` interpolation?

I would love if you had sources defining somewhere an accepted standard, but I'm not sure there is, and in this case, `sklearn` (which kind of is a standard of its own in the CS stats world) should explicitly state (and justify) its definition. I think there is small lack at the moment.

Admitedly, it remains an edge case (at least when comparing the `previous` and `linear` approaches) and we may consider it unimportant, I would understand it.

Labels: Bug
[BUG] `roc_auc_score` is wrong (edge case)

### Describe the bug

Hi,

It seems that on the edge case when there are equal scores, [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#roc-auc-score) actually makes a wrong comuptation:
```pycon
>>> roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
0.375
>>> # Expected 0.25 or 0.5, see below.
```
On would expect either `0.5` or `0.25` depending on whether or not the area is computed by interpolating to the convex hull (which makes sense with probabilistic mixtures for the ROC curve) or not.

I tried illustrating this example below (keep in mind that this behaviour is *speculated* by myself, I didn't go through any of `sklearn` code).

![roc_auc_bug](https://github.com/scikit-learn/scikit-learn/assets/114467748/4a902ca2-fc23-4341-b8b9-6234a9f40fd8)

Any feedback is welcome.
Cheers!

### Steps/Code to Reproduce

```pycon
from sklearn.metrics import roc_auc_score
roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
```

### Expected Results

`0.25` or `0.5`

### Actual Results

`0.375`

### Versions

```shell
1.5.0
```

COMMENT:
The plot in wikipedia https://en.wikipedia.org/wiki/Receiver_operating_characteristic has linear slope sections and is non-convex so it's probably using the same naive linear interpolation as scikit-learn.

And I explained before why I don't think `previous` interpolation is a valid choice (for the case of the constant predictor).

ROCCH seems to be getting traction on the other hand but it's getting traction under that specific name.

> and we may consider it unimportant, I would understand it.

The difference between `previous` and `linear` is probably only visible when the number of tied predictions is very large compared to the dataset size.

The ROCCH alternative on the other hand seems to have profound implications that are probably worth exploring, for instance:

- Technical Note: PAV and the ROC Convex Hull
  http://www.niculescu-mizil.org/papers/PAV-ROCCH.pdf

- Classifier Calibration with ROC-Regularized Isotonic Regression
  https://proceedings.mlr.press/v238/berta24a.html

Note that I have not yet carefully read this papers.

Labels: Bug
[BUG] `roc_auc_score` is wrong (edge case)

### Describe the bug

Hi,

It seems that on the edge case when there are equal scores, [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#roc-auc-score) actually makes a wrong comuptation:
```pycon
>>> roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
0.375
>>> # Expected 0.25 or 0.5, see below.
```
On would expect either `0.5` or `0.25` depending on whether or not the area is computed by interpolating to the convex hull (which makes sense with probabilistic mixtures for the ROC curve) or not.

I tried illustrating this example below (keep in mind that this behaviour is *speculated* by myself, I didn't go through any of `sklearn` code).

![roc_auc_bug](https://github.com/scikit-learn/scikit-learn/assets/114467748/4a902ca2-fc23-4341-b8b9-6234a9f40fd8)

Any feedback is welcome.
Cheers!

### Steps/Code to Reproduce

```pycon
from sklearn.metrics import roc_auc_score
roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
```

### Expected Results

`0.25` or `0.5`

### Actual Results

`0.375`

### Versions

```shell
1.5.0
```

COMMENT:
So I would be in favor of closing this bug report as "not planned" and instead open a feature request for ROCCH-based evaluation tools (curve + AUC score).

Related: there was a very old PR about ROCCH-based calibration in the past but it was never properly reviewed:

- https://github.com/scikit-learn/scikit-learn/pull/4822

Labels: Bug
[BUG] `roc_auc_score` is wrong (edge case)

### Describe the bug

Hi,

It seems that on the edge case when there are equal scores, [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#roc-auc-score) actually makes a wrong comuptation:
```pycon
>>> roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
0.375
>>> # Expected 0.25 or 0.5, see below.
```
On would expect either `0.5` or `0.25` depending on whether or not the area is computed by interpolating to the convex hull (which makes sense with probabilistic mixtures for the ROC curve) or not.

I tried illustrating this example below (keep in mind that this behaviour is *speculated* by myself, I didn't go through any of `sklearn` code).

![roc_auc_bug](https://github.com/scikit-learn/scikit-learn/assets/114467748/4a902ca2-fc23-4341-b8b9-6234a9f40fd8)

Any feedback is welcome.
Cheers!

### Steps/Code to Reproduce

```pycon
from sklearn.metrics import roc_auc_score
roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
```

### Expected Results

`0.25` or `0.5`

### Actual Results

`0.375`

### Versions

```shell
1.5.0
```

COMMENT:
Thanks. Feel free to open a PR to implement the ROCCH option.

For the record, if someone comes up with a good reference in the literature about `previous` vs `linear` w.r.t. the canonical definition of a ROC curve in an ML context, we should definitely consider reopening that argument.

Labels: Bug
[BUG] `roc_auc_score` is wrong (edge case)

### Describe the bug

Hi,

It seems that on the edge case when there are equal scores, [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#roc-auc-score) actually makes a wrong comuptation:
```pycon
>>> roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
0.375
>>> # Expected 0.25 or 0.5, see below.
```
On would expect either `0.5` or `0.25` depending on whether or not the area is computed by interpolating to the convex hull (which makes sense with probabilistic mixtures for the ROC curve) or not.

I tried illustrating this example below (keep in mind that this behaviour is *speculated* by myself, I didn't go through any of `sklearn` code).

![roc_auc_bug](https://github.com/scikit-learn/scikit-learn/assets/114467748/4a902ca2-fc23-4341-b8b9-6234a9f40fd8)

Any feedback is welcome.
Cheers!

### Steps/Code to Reproduce

```pycon
from sklearn.metrics import roc_auc_score
roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
```

### Expected Results

`0.25` or `0.5`

### Actual Results

`0.375`

### Versions

```shell
1.5.0
```

COMMENT:
I thought of something that might benefit the discussion: the **probabilistic interpretation of AUC**.

Consider random variables $X_0$ (_resp._ $X_1$), corresponding to the score of a uniformly random negative (_resp._ positive) sample, drawn from the dataset used to plot the ROC curve. In the convention where positive instances should have higher scores (_e.g._ my opening message), the usual AUC interpretation is $\text{AUC} = \mathbb{P}(X_1 > X_0)$ or $\text{AUC} = \mathbb{P}(X_1 \geqslant X_0)$ (Wikipedia is inconsistent on this as for example, they use the former in the first paragraph of [Area under the curve](https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve) and they use the latter in [probabilistic interpretation of AUC](https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Probabilistic_interpretation). It doesn't matter in the case of continuous probability distributions).

Then, relative to our previous discussions, it can be easily shown that:
- `previous` (pessimistic) interpolation yields $\text{AUC} = \mathbb{P}(X_1 > X_0)$;
- `next` (optimistic) interpolation yields $\text{AUC} = \mathbb{P}(X_1 \geqslant X_0)$;
- `linear` interpolation yields $\text{AUC} = \frac{1}{2}\big(\mathbb{P}(X_1\geqslant X_0) + \mathbb{P}(X_1 > X_0)\big)$.

Labels: Bug
[BUG] `roc_auc_score` is wrong (edge case)

### Describe the bug

Hi,

It seems that on the edge case when there are equal scores, [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#roc-auc-score) actually makes a wrong comuptation:
```pycon
>>> roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
0.375
>>> # Expected 0.25 or 0.5, see below.
```
On would expect either `0.5` or `0.25` depending on whether or not the area is computed by interpolating to the convex hull (which makes sense with probabilistic mixtures for the ROC curve) or not.

I tried illustrating this example below (keep in mind that this behaviour is *speculated* by myself, I didn't go through any of `sklearn` code).

![roc_auc_bug](https://github.com/scikit-learn/scikit-learn/assets/114467748/4a902ca2-fc23-4341-b8b9-6234a9f40fd8)

Any feedback is welcome.
Cheers!

### Steps/Code to Reproduce

```pycon
from sklearn.metrics import roc_auc_score
roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
```

### Expected Results

`0.25` or `0.5`

### Actual Results

`0.375`

### Versions

```shell
1.5.0
```

COMMENT:
In case it is helpful, [this piece](https://www.sciencedirect.com/science/article/pii/S016786550500303X) seems to recommend the linear interpolation option (see section 5).

Labels: Bug
⚠️ CI failed on Wheel builder (last failure: Jun 13, 2024) ⚠️

**CI failed on [Wheel builder](https://github.com/scikit-learn/scikit-learn/actions/runs/9493360666)** (Jun 13, 2024)

COMMENT:
`OSError: GetModuleFileNameEx failed` coming from threadpoolctl on Windows Python 3.12, see [build log](https://github.com/scikit-learn/scikit-learn/actions/runs/9493360666/job/26161928493) cc @jeremiedbb in case he has already seen this kind of thing. I have restarted the build to see if it persists.

The error was:
```
  + docker container run -e SKLEARN_SKIP_NETWORK_TESTS=1 --rm scikit-learn/minimal-windows powershell -Command 'pytest --pyargs sklearn'
  ============================= test session starts =============================
  platform win32 -- Python 3.12.4, pytest-8.2.2, pluggy-1.5.0
  rootdir: C:\
  collected 0 items / 1 error
  
  =================================== ERRORS ====================================
  ______________ ERROR collecting Python/Lib/site-packages/sklearn ______________
  Python\Lib\importlib\__init__.py:90: in import_module
      return _bootstrap._gcd_import(name[level:], package, level)
  <frozen importlib._bootstrap>:1387: in _gcd_import
      ???
  <frozen importlib._bootstrap>:1360: in _find_and_load
      ???
  <frozen importlib._bootstrap>:1310: in _find_and_load_unlocked
      ???
  <frozen importlib._bootstrap>:488: in _call_with_frames_removed
      ???
  <frozen importlib._bootstrap>:1387: in _gcd_import
      ???
  <frozen importlib._bootstrap>:1360: in _find_and_load
      ???
  <frozen importlib._bootstrap>:1331: in _find_and_load_unlocked
      ???
  <frozen importlib._bootstrap>:935: in _load_unlocked
      ???
  <frozen importlib._bootstrap_external>:995: in exec_module
      ???
  <frozen importlib._bootstrap>:488: in _call_with_frames_removed
      ???
  Python\Lib\site-packages\sklearn\__init__.py:154: in <module>
      _threadpool_controller = ThreadpoolController()
  Python\Lib\site-packages\threadpoolctl.py:818: in __init__
      self._load_libraries()
  Python\Lib\site-packages\threadpoolctl.py:972: in _load_libraries
      self._find_libraries_with_enum_process_module_ex()
  Python\Lib\site-packages\threadpoolctl.py:1096: in _find_libraries_with_enum_process_module_ex
      raise OSError("GetModuleFileNameEx failed")
  E   OSError: GetModuleFileNameEx failed
  =========================== short test summary info ===========================
  ERROR Python/Lib/site-packages/sklearn - OSError: GetModuleFileNameEx failed
  !!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!
  ============================== 1 error in 1.92s ===============================
```

Labels: Needs Triage
⚠️ CI failed on Wheel builder (last failure: Jun 13, 2024) ⚠️

**CI failed on [Wheel builder](https://github.com/scikit-learn/scikit-learn/actions/runs/9493360666)** (Jun 13, 2024)

COMMENT:
I've never seen this function fail. If it's confirmed it will be a fun debugging because it's 60 lines of system level windows specific code that we took from somewhere else :smile:

Labels: Needs Triage
⚠️ CI failed on Wheel builder (last failure: Jun 13, 2024) ⚠️

**CI failed on [Wheel builder](https://github.com/scikit-learn/scikit-learn/actions/runs/9493360666)** (Jun 13, 2024)

COMMENT:
Let's hope it was a glitch of some sort :wink:. Edit, looks like it was since the tests have started running: https://github.com/scikit-learn/scikit-learn/actions/runs/9493360666/job/26172291328

I have seen one [SO question](https://stackoverflow.com/q/78377863) mentioning the issue, no answer obviously ...

I have noticed that dlllist uses `ctypes.get_last_error()` to maybe give a bit more details (probably quite cryptic) when these functions fail, see https://github.com/WardBrian/dllist/blob/main/dllist/windows.py, maybe something to consider doing in threadpoolctl?

Labels: Needs Triage
⚠️ CI failed on Wheel builder (last failure: Jun 13, 2024) ⚠️

**CI failed on [Wheel builder](https://github.com/scikit-learn/scikit-learn/actions/runs/9493360666)** (Jun 13, 2024)

COMMENT:
> I have noticed that dlllist uses ctypes.get_last_error() to maybe give a bit more details (probably quite cryptic) when these functions fail, see https://github.com/WardBrian/dllist/blob/main/dllist/windows.py, maybe something to consider doing in threadpoolctl?

+1

Labels: Needs Triage
⚠️ CI failed on Wheel builder (last failure: Jun 13, 2024) ⚠️

**CI failed on [Wheel builder](https://github.com/scikit-learn/scikit-learn/actions/runs/9493360666)** (Jun 13, 2024)

COMMENT:
## CI is no longer failing! ✅

[Successful run](https://github.com/scikit-learn/scikit-learn/actions/runs/9493360666) on Jun 13, 2024

Labels: Needs Triage
⚠️ CI failed on Wheel builder (last failure: Jun 13, 2024) ⚠️

**CI failed on [Wheel builder](https://github.com/scikit-learn/scikit-learn/actions/runs/9493360666)** (Jun 13, 2024)

COMMENT:
The last CI run was fine, so it looks like this was some kind of temporary glitch ...

Labels: Needs Triage
Maintenance: GPU CI follow up work

This is a follow up to #24491. As we are starting touse the GPU CI workflow we are noticing things we don't like and things which are missing. Let's collect them in this issue as a way to keep track of them.

Things to add:
* [ ] comment (or other notification) in a PR to show the workflow's status
  * use https://github.com/myrotvorets/set-commit-status-action, description in [discord](https://discord.com/channels/731163543038197871/1244783982516899880/1248132914328371231) via @lesteve 
  * use a updating comment in the PR with a bit of explanation for contributors regarding when the workflow runs, etc
  * mention https://gist.github.com/EdAbati/ff3bdc06bafeb92452b3740686cc8d7c via @EdAbati in the comment so people can debug things without needing the workflow to run
* [ ] keep an eye on the costs (currently $50/month spending limit)

COMMENT:
I'll close this for now. Most of the work is done and the CI is being used. Let's create more focussed issues if we need them

Labels: Build / CI, Array API
tree.plot_tree() method takes too long then crashes 1.2.2

### Describe the bug

Hello Everyone,

So i'm training a decision tree model on a trainng set of about 200k rows and 8 cols, the training itself doesn't take long but when i try to visualize the tree using tree.plot_tree(), it doesn't work and i wait too long for it, untill it crashes.

Here is my code:

```python
plt.figure(figsize=(20,10))
plot_tree(model, feature_names=X_train.columns, class_names=['True', 'False'], filled=True)
plt.show()
```

It is a binary classification, so i put the ['True', 'False'], is there something wrong with my code, or is the data just too large ?

Thanks a lot and have a nice day 


### Steps/Code to Reproduce

```python
# Import necessary libraries

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier, plot_tree
import matplotlib.pyplot as plt

# Step 1: Load the Iris dataset
iris = load_iris()
X, y = iris.data, iris.target

# Step 2: Train a decision tree classifier
clf = DecisionTreeClassifier()
clf.fit(X, y)

# Step 3: Visualize the decision tree
plt.figure(figsize=(20, 10))  # Set the figure size for better visibility
plot_tree(clf, filled=True, feature_names=iris.feature_names, class_names=iris.target_names)
```
### Expected Results

A tree visualization 

### Actual Results

Takes too long, and then kernel crash

### Versions

```shell
Version 1.2.2
```

COMMENT:
I cannot reproduce using the example that you provide.
It works as expected with the latest scikit-learn.

Labels: Bug, Needs Info