Talip7/scikit-learn-issues-multilabel huggingface dataset:
[BUG] `roc_auc_score` is wrong (edge case)

### Describe the bug

Hi,

It seems that on the edge case when there are equal scores, [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#roc-auc-score) actually makes a wrong comuptation:
```pycon
>>> roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
0.375
>>> # Expected 0.25 or 0.5, see below.
```
On would expect either `0.5` or `0.25` depending on whether or not the area is computed by interpolating to the convex hull (which makes sense with probabilistic mixtures for the ROC curve) or not.

I tried illustrating this example below (keep in mind that this behaviour is *speculated* by myself, I didn't go through any of `sklearn` code).

![roc_auc_bug](https://github.com/scikit-learn/scikit-learn/assets/114467748/4a902ca2-fc23-4341-b8b9-6234a9f40fd8)

Any feedback is welcome.
Cheers!

### Steps/Code to Reproduce

```pycon
from sklearn.metrics import roc_auc_score
roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
```

### Expected Results

`0.25` or `0.5`

### Actual Results

`0.375`

### Versions

```shell
1.5.0
```

COMMENT:
Sure @ogrisel!

```pycon
>>> labels = [1, 0, 0, 1]
>>> scores = [2, 5, 10, 10]
>>> fpr, tpr, thresholds = sklearn.metrics.roc_curve(labels, scores)
>>> _ = sklearn.metrics.RocCurveDisplay(fpr=fpr, tpr=tpr).plot()
>>> plt.show()
```

![sklearn_roc](https://github.com/scikit-learn/scikit-learn/assets/114467748/694f7808-7933-4dd1-bb34-bf3399de2c08)

<sub><sup>_It seems I made a small mistake regarding conventions since the curve is "inverted" (the other convention I used can be recovered by negating both `labels` and `scores`) but it's not important._</sup></sub>

My takeaway is that:
- `sklearn` indeed correctly identifies the unattainable `(FPR, TPR)` point due to two last scores being the same,
- `sklearn` interpolates linearly between every points, which I think is a weird behaviour _statistically speaking_. In my opinion, it should either interpolate to the previous point (center illustration above) or to the convex hull (right illustration above).

Complementary data :

```pycon
>>> thresholds
array([11, 10,  5,  2])
>>> fpr
array([0. , 0.5, 1. , 1. ])
>>> tpr
array([0. , 0.5, 0.5, 1. ])
```

Labels: Bug
[BUG] `roc_auc_score` is wrong (edge case)

### Describe the bug

Hi,

It seems that on the edge case when there are equal scores, [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#roc-auc-score) actually makes a wrong comuptation:
```pycon
>>> roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
0.375
>>> # Expected 0.25 or 0.5, see below.
```
On would expect either `0.5` or `0.25` depending on whether or not the area is computed by interpolating to the convex hull (which makes sense with probabilistic mixtures for the ROC curve) or not.

I tried illustrating this example below (keep in mind that this behaviour is *speculated* by myself, I didn't go through any of `sklearn` code).

![roc_auc_bug](https://github.com/scikit-learn/scikit-learn/assets/114467748/4a902ca2-fc23-4341-b8b9-6234a9f40fd8)

Any feedback is welcome.
Cheers!

### Steps/Code to Reproduce

```pycon
from sklearn.metrics import roc_auc_score
roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
```

### Expected Results

`0.25` or `0.5`

### Actual Results

`0.375`

### Versions

```shell
1.5.0
```

COMMENT:
> - `sklearn` interpolates linearly between every points, which I think is a weird behaviour _statistically speaking_. In my opinion, it should either interpolate to the previous point (center illustration above) or to the convex hull (right illustration above).

I just want to clarify this. The ROC curve is meant to inform an operator what `TPR` they can expect given a chosen `FPR`.
- The `TPR` obtained by interpolating with the `previous` point (in [scipy](https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.interp1d.html#scipy-interpolate-interp1d)'s legacy terms -- see `kind` keyword) corresponds to simply thresholding the classifier with an appropriate threshold.
- But also, the `TPR` obtained by interpolating to the convex hull can always be attained by thresholding stochastically using 2 thresholds and an adujsted coinflip. _(careful, this is true for the ROC curve, but not the PR curve for example, at least not in this form)_

Thus, I think the curve computed by `sklearn` doesn't really make sense since it allows interpolation between points but doesn't extend it to the whole convex hull.

TLDR: The ROC curve should either be a step function or a convex function.

Labels: Bug
[BUG] `roc_auc_score` is wrong (edge case)

### Describe the bug

Hi,

It seems that on the edge case when there are equal scores, [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#roc-auc-score) actually makes a wrong comuptation:
```pycon
>>> roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
0.375
>>> # Expected 0.25 or 0.5, see below.
```
On would expect either `0.5` or `0.25` depending on whether or not the area is computed by interpolating to the convex hull (which makes sense with probabilistic mixtures for the ROC curve) or not.

I tried illustrating this example below (keep in mind that this behaviour is *speculated* by myself, I didn't go through any of `sklearn` code).

![roc_auc_bug](https://github.com/scikit-learn/scikit-learn/assets/114467748/4a902ca2-fc23-4341-b8b9-6234a9f40fd8)

Any feedback is welcome.
Cheers!

### Steps/Code to Reproduce

```pycon
from sklearn.metrics import roc_auc_score
roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
```

### Expected Results

`0.25` or `0.5`

### Actual Results

`0.375`

### Versions

```shell
1.5.0
```

COMMENT:
Thanks for the update. This might indeed be a bug in our choice of interpolation for this curve. Do you have good literature references at hand to back your suggestion for different interpolation choices?

Labels: Bug
[BUG] `roc_auc_score` is wrong (edge case)

### Describe the bug

Hi,

It seems that on the edge case when there are equal scores, [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#roc-auc-score) actually makes a wrong comuptation:
```pycon
>>> roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
0.375
>>> # Expected 0.25 or 0.5, see below.
```
On would expect either `0.5` or `0.25` depending on whether or not the area is computed by interpolating to the convex hull (which makes sense with probabilistic mixtures for the ROC curve) or not.

I tried illustrating this example below (keep in mind that this behaviour is *speculated* by myself, I didn't go through any of `sklearn` code).

![roc_auc_bug](https://github.com/scikit-learn/scikit-learn/assets/114467748/4a902ca2-fc23-4341-b8b9-6234a9f40fd8)

Any feedback is welcome.
Cheers!

### Steps/Code to Reproduce

```pycon
from sklearn.metrics import roc_auc_score
roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
```

### Expected Results

`0.25` or `0.5`

### Actual Results

`0.375`

### Versions

```shell
1.5.0
```

COMMENT:
I don't really have a specific source in mind as this is a rather "elementary" consideration.

I did find this paragraph on the [ROC wiki](https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve):
> It is also common to calculate the Area Under the ROC Convex Hull (ROC AUCH = ROCH AUC) as any point on the line segment between two prediction results can be achieved by randomly using one or the other system with probabilities proportional to the relative length of the opposite component of the segment [[source]](https://link.springer.com/article/10.1023/A:1007601015854#preview).

Also, [this paper](https://bmva-archive.org.uk/bmvc/1998/pdf/p082.pdf) (Scott, Niranjan and Prager -- 1998) studies the convex hull of ROCs for  multiple classifier, which is equivalent since one classifier can be thought of as many "single point" classifiers (one for every different relevant threshold).

Labels: Bug
[BUG] `roc_auc_score` is wrong (edge case)

### Describe the bug

Hi,

It seems that on the edge case when there are equal scores, [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#roc-auc-score) actually makes a wrong comuptation:
```pycon
>>> roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
0.375
>>> # Expected 0.25 or 0.5, see below.
```
On would expect either `0.5` or `0.25` depending on whether or not the area is computed by interpolating to the convex hull (which makes sense with probabilistic mixtures for the ROC curve) or not.

I tried illustrating this example below (keep in mind that this behaviour is *speculated* by myself, I didn't go through any of `sklearn` code).

![roc_auc_bug](https://github.com/scikit-learn/scikit-learn/assets/114467748/4a902ca2-fc23-4341-b8b9-6234a9f40fd8)

Any feedback is welcome.
Cheers!

### Steps/Code to Reproduce

```pycon
from sklearn.metrics import roc_auc_score
roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
```

### Expected Results

`0.25` or `0.5`

### Actual Results

`0.375`

### Versions

```shell
1.5.0
```

COMMENT:
I think the problem does not come from linear interpolation but rather from the fact that we miss a synthetic points compared to your diagrams. Here are the FPR/TPR/thresholds values computed by the scikit-learn roc_curve:

```python
>>> fpr
array([0. , 0.5, 1. , 1. ])
>>> tpr
array([0. , 0.5, 0.5, 1. ])
>>> thresholds
array([inf, 10.,  5.,  2.])
```

while your plots show:

```python
>>> fpr
array([0. , 0.5, 0.5, 1. , 1. ])
>>> tpr
array([0. , 0., 0.5, 0.5, 1. ])
```

But I am not sure what would be the threshold value for that second point.

Labels: Bug
[BUG] `roc_auc_score` is wrong (edge case)

### Describe the bug

Hi,

It seems that on the edge case when there are equal scores, [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#roc-auc-score) actually makes a wrong comuptation:
```pycon
>>> roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
0.375
>>> # Expected 0.25 or 0.5, see below.
```
On would expect either `0.5` or `0.25` depending on whether or not the area is computed by interpolating to the convex hull (which makes sense with probabilistic mixtures for the ROC curve) or not.

I tried illustrating this example below (keep in mind that this behaviour is *speculated* by myself, I didn't go through any of `sklearn` code).

![roc_auc_bug](https://github.com/scikit-learn/scikit-learn/assets/114467748/4a902ca2-fc23-4341-b8b9-6234a9f40fd8)

Any feedback is welcome.
Cheers!

### Steps/Code to Reproduce

```pycon
from sklearn.metrics import roc_auc_score
roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
```

### Expected Results

`0.25` or `0.5`

### Actual Results

`0.375`

### Versions

```shell
1.5.0
```

COMMENT:
Note: one could consider to add ROC Convex Hull curve and AUC to scikit-learn but I have the feeling that this is a distinct feature request compared to this specific edge case bug report.

Labels: Bug
[BUG] `roc_auc_score` is wrong (edge case)

### Describe the bug

Hi,

It seems that on the edge case when there are equal scores, [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#roc-auc-score) actually makes a wrong comuptation:
```pycon
>>> roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
0.375
>>> # Expected 0.25 or 0.5, see below.
```
On would expect either `0.5` or `0.25` depending on whether or not the area is computed by interpolating to the convex hull (which makes sense with probabilistic mixtures for the ROC curve) or not.

I tried illustrating this example below (keep in mind that this behaviour is *speculated* by myself, I didn't go through any of `sklearn` code).

![roc_auc_bug](https://github.com/scikit-learn/scikit-learn/assets/114467748/4a902ca2-fc23-4341-b8b9-6234a9f40fd8)

Any feedback is welcome.
Cheers!

### Steps/Code to Reproduce

```pycon
from sklearn.metrics import roc_auc_score
roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
```

### Expected Results

`0.25` or `0.5`

### Actual Results

`0.375`

### Versions

```shell
1.5.0
```

COMMENT:
Hum, this might not be only a problem with a missing point at the hypothetical second position. Looking at that example:

https://scikit-learn.org/stable/auto_examples/miscellaneous/plot_roc_curve_visualization_api.html#sphx-glr-auto-examples-miscellaneous-plot-roc-curve-visualization-api-py

![image](https://github.com/scikit-learn/scikit-learn/assets/89061/0461fb6a-3961-45e7-adf8-8314cd4acc7e)

we see that the curve for the SVC model displays the expected step-wise structure while we do see linear slopes for some thresholds of the RF model.

Labels: Bug
[BUG] `roc_auc_score` is wrong (edge case)

### Describe the bug

Hi,

It seems that on the edge case when there are equal scores, [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#roc-auc-score) actually makes a wrong comuptation:
```pycon
>>> roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
0.375
>>> # Expected 0.25 or 0.5, see below.
```
On would expect either `0.5` or `0.25` depending on whether or not the area is computed by interpolating to the convex hull (which makes sense with probabilistic mixtures for the ROC curve) or not.

I tried illustrating this example below (keep in mind that this behaviour is *speculated* by myself, I didn't go through any of `sklearn` code).

![roc_auc_bug](https://github.com/scikit-learn/scikit-learn/assets/114467748/4a902ca2-fc23-4341-b8b9-6234a9f40fd8)

Any feedback is welcome.
Cheers!

### Steps/Code to Reproduce

```pycon
from sklearn.metrics import roc_auc_score
roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
```

### Expected Results

`0.25` or `0.5`

### Actual Results

`0.375`

### Versions

```shell
1.5.0
```

COMMENT:
Thank you for your answers. The synthetic point you are referring to does not exist in the list for the exact reason that it is not associated with a threshold (you can observe in the illustration of the opening message that this point -- well, the point `(1, 0.5)` -- does not appear in blue).

This misconception comes from the fact that we are talking about a "curve" when In fact only a finite number of points in the graph are attainable with a single threshold.

The easiest way to "fill the gaps" between these points is to interpolate with a step function, and the most optimal way to fill it is by interpolating linearly between the points of the Pareto frontier (which corresponds to using stochastic thresholding).

Now in my opinion, the question that needs to be answered to solve this issue is "what is our correct definition of the ROC?". And the current implementation answers "a weird mix in the middle", which is, again in my opinion, the only incorrect answer possible from a statistics point of view.

Labels: Bug
[BUG] `roc_auc_score` is wrong (edge case)

### Describe the bug

Hi,

It seems that on the edge case when there are equal scores, [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#roc-auc-score) actually makes a wrong comuptation:
```pycon
>>> roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
0.375
>>> # Expected 0.25 or 0.5, see below.
```
On would expect either `0.5` or `0.25` depending on whether or not the area is computed by interpolating to the convex hull (which makes sense with probabilistic mixtures for the ROC curve) or not.

I tried illustrating this example below (keep in mind that this behaviour is *speculated* by myself, I didn't go through any of `sklearn` code).

![roc_auc_bug](https://github.com/scikit-learn/scikit-learn/assets/114467748/4a902ca2-fc23-4341-b8b9-6234a9f40fd8)

Any feedback is welcome.
Cheers!

### Steps/Code to Reproduce

```pycon
from sklearn.metrics import roc_auc_score
roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
```

### Expected Results

`0.25` or `0.5`

### Actual Results

`0.375`

### Versions

```shell
1.5.0
```

COMMENT:
> we see that the curve for the SVC model displays the expected step-wise structure while we do see linear slopes for some thresholds of the RF model.

Thanks for the complementary example. In the current implementation, "linear slopes" arise when two consecutive (when varying the threshold) `(FPR, TPR)` couples have both different `FPR` **and** different `TPR`. This can only occur when there are multiple predictions with the exact same score and different labels (making it impossible to separate them by thresholding). When there ar no such instances, the ROC will still be step-wise.

So I think the example you provided is probably showing the same behaviour as my MWE of the beginning. What do you think?

Labels: Bug
[BUG] `roc_auc_score` is wrong (edge case)

### Describe the bug

Hi,

It seems that on the edge case when there are equal scores, [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#roc-auc-score) actually makes a wrong comuptation:
```pycon
>>> roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
0.375
>>> # Expected 0.25 or 0.5, see below.
```
On would expect either `0.5` or `0.25` depending on whether or not the area is computed by interpolating to the convex hull (which makes sense with probabilistic mixtures for the ROC curve) or not.

I tried illustrating this example below (keep in mind that this behaviour is *speculated* by myself, I didn't go through any of `sklearn` code).

![roc_auc_bug](https://github.com/scikit-learn/scikit-learn/assets/114467748/4a902ca2-fc23-4341-b8b9-6234a9f40fd8)

Any feedback is welcome.
Cheers!

### Steps/Code to Reproduce

```pycon
from sklearn.metrics import roc_auc_score
roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
```

### Expected Results

`0.25` or `0.5`

### Actual Results

`0.375`

### Versions

```shell
1.5.0
```

COMMENT:
Here is a minimal case where the linear slopes appear not on the extremes of the curve:

```python
from sklearn.metrics import RocCurveDisplay

roc_disp = RocCurveDisplay.from_predictions([0, 0, 0, 1, 1], [0, 1, 2, 1, 3])
print(roc_disp.fpr)
print(roc_disp.tpr)
```
![image](https://github.com/scikit-learn/scikit-learn/assets/89061/72e6f081-3f4a-4d6e-bab0-65cfd0dc3907)

```
[0. 0. 0.33... 0.66..., 1.]
[0.  0.5 0.5 1.  1. ]
```
Basically we need mis-ranked predictions with exact ties, which are more likely to happen with tree-based models because of their piecewise constant prediction function.

Labels: Bug
[BUG] `roc_auc_score` is wrong (edge case)

### Describe the bug

Hi,

It seems that on the edge case when there are equal scores, [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#roc-auc-score) actually makes a wrong comuptation:
```pycon
>>> roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
0.375
>>> # Expected 0.25 or 0.5, see below.
```
On would expect either `0.5` or `0.25` depending on whether or not the area is computed by interpolating to the convex hull (which makes sense with probabilistic mixtures for the ROC curve) or not.

I tried illustrating this example below (keep in mind that this behaviour is *speculated* by myself, I didn't go through any of `sklearn` code).

![roc_auc_bug](https://github.com/scikit-learn/scikit-learn/assets/114467748/4a902ca2-fc23-4341-b8b9-6234a9f40fd8)

Any feedback is welcome.
Cheers!

### Steps/Code to Reproduce

```pycon
from sklearn.metrics import roc_auc_score
roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
```

### Expected Results

`0.25` or `0.5`

### Actual Results

`0.375`

### Versions

```shell
1.5.0
```

COMMENT:
In the absence of exact tied misranked predictions, the ROC points would be always excactly arranged as consecutive steps: either a non-zero increment in FPR or in TPR but not both at the same time, hence the step-wise look of common ROC curves on small datasets.

In this case the choice of the naive interpolation (`kind="previous"` or `kind="linear"`) would have no impact, and indeed `kind="previous"` might be more valid in the presence of misranked ties.

Convex / statistical interpolation might also be interesting to add but feels like a lot more work and should probably be treated as an enhancement rather than a bug fix.

Labels: Bug
[BUG] `roc_auc_score` is wrong (edge case)

### Describe the bug

Hi,

It seems that on the edge case when there are equal scores, [`roc_auc_score`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html#roc-auc-score) actually makes a wrong comuptation:
```pycon
>>> roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
0.375
>>> # Expected 0.25 or 0.5, see below.
```
On would expect either `0.5` or `0.25` depending on whether or not the area is computed by interpolating to the convex hull (which makes sense with probabilistic mixtures for the ROC curve) or not.

I tried illustrating this example below (keep in mind that this behaviour is *speculated* by myself, I didn't go through any of `sklearn` code).

![roc_auc_bug](https://github.com/scikit-learn/scikit-learn/assets/114467748/4a902ca2-fc23-4341-b8b9-6234a9f40fd8)

Any feedback is welcome.
Cheers!

### Steps/Code to Reproduce

```pycon
from sklearn.metrics import roc_auc_score
roc_auc_score([1, 0, 0, 1], [2, 5, 10, 10])
```

### Expected Results

`0.25` or `0.5`

### Actual Results

`0.375`

### Versions

```shell
1.5.0
```

COMMENT:
> and indeed kind="previous" might be more valid in the presence of misranked ties.

Actually I am not so sure: if you consider that ties should be randomly broken, then the linear interpolation might better represent that. For instance let's randomly break ties by adding machine-level precision noise to the predictions:

```python
import numpy as np
from sklearn.metrics import RocCurveDisplay
from sklearn.metrics import roc_auc_score
import matplotlib.pyplot as plt

fig, ax = plt.subplots()
y_observed = np.asarray([0, 0, 0, 1, 1])
y_predicted_with_exact_ties = np.asarray([0, 1, 2, 1, 3])
for i in range(10):
    y_predicted = y_predicted_with_exact_ties + np.random.randn(y_observed.size) * 1e-12
    roc_disp = RocCurveDisplay.from_predictions(y_observed, y_predicted, ax=ax)

auc_scores = []
for i in range(10000):
    y_predicted = y_predicted_with_exact_ties + np.random.randn(y_observed.size) * 1e-12
    auc_scores.append(roc_auc_score(y_observed, y_predicted))

print(np.mean(auc_scores))
```
```
0.7508166666666668
```
![image](https://github.com/scikit-learn/scikit-learn/assets/89061/b0df133e-fc60-4ee0-940c-06973faa8c00)

You see that half the results have an AUC of 0.67 and the other half has 0.83, so the ROC AUC of 0.75 of the linear interpolation is actually a more neutral way to break tied predictions than using the constant piecewise interpolation.

Labels: Bug