Talip7/scikit-learn-issues-multilabel huggingface dataset:
average_precision_score not working as expected

### Describe the bug

When compute AP with average_precision_score, I get unexpected results. The y_scores (output from the models) are very low for positive samples, so my AP should be very low. Instead I get a perfect 1.0 AP score.

### Steps/Code to Reproduce

```python
from sklearn.metrics import average_precision_score

# labels
y_true = [1, 1, 1, 1, 1]  # 5 positives
# Predicted scores
y_scores = [0.1, 0.3, 0.1, 0.2, 0.1]  # Model's confidence in predictions

# Calculate average precision
average_precision = average_precision_score(y_true, y_scores)
print("AP score:", average_precision)
```

### Expected Results

An AP score close to 0.

### Actual Results

AP score: 1.0

### Versions

```shell
System:
    python: 3.9.19 (main, May  6 2024, 19:43:03)  [GCC 11.2.0]
executable: /bin/python
   machine: Linux-5.15.0-121-generic-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.5.2
          pip: 24.2
   setuptools: 75.1.0
        numpy: 1.26.4
        scipy: 1.13.1
       Cython: None
       pandas: 1.4.0
   matplotlib: 3.5.3
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 32
         prefix: libopenblas
       filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: SkylakeX

       user_api: blas
   internal_api: openblas
    num_threads: 32
         prefix: libopenblas
       filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/scipy.libs/libopenblasp-r0-01191904.3.27.so
        version: 0.3.27
threading_layer: pthreads
   architecture: SkylakeX

       user_api: openmp
   internal_api: openmp
    num_threads: 32
         prefix: libgomp
       filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

COMMENT:
> This would be the score of a DummyClassifier(strategy="constant", constant=1). It means that any predictor providing a score under this threshold do worse that a rule that does not learn any statistical association between the data and the target. So this is reasonable to me.

Not really because my application is Object Detection, where AP (or mAP) is the main metric used by most papers and benchmarks (e.g. COCO: https://cocodataset.org/#detection-eval).
In that case you can't predict all positives (or you would have an infinite number of false positives). That's what I meant by saying that when the model has 0 confidence in my example, it means it missed completely the detection, so it should never be counted as a positive no matter the threshold. Recall can't be above 0.55 in this case.

Labels: Bug
average_precision_score not working as expected

### Describe the bug

When compute AP with average_precision_score, I get unexpected results. The y_scores (output from the models) are very low for positive samples, so my AP should be very low. Instead I get a perfect 1.0 AP score.

### Steps/Code to Reproduce

```python
from sklearn.metrics import average_precision_score

# labels
y_true = [1, 1, 1, 1, 1]  # 5 positives
# Predicted scores
y_scores = [0.1, 0.3, 0.1, 0.2, 0.1]  # Model's confidence in predictions

# Calculate average precision
average_precision = average_precision_score(y_true, y_scores)
print("AP score:", average_precision)
```

### Expected Results

An AP score close to 0.

### Actual Results

AP score: 1.0

### Versions

```shell
System:
    python: 3.9.19 (main, May  6 2024, 19:43:03)  [GCC 11.2.0]
executable: /bin/python
   machine: Linux-5.15.0-121-generic-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.5.2
          pip: 24.2
   setuptools: 75.1.0
        numpy: 1.26.4
        scipy: 1.13.1
       Cython: None
       pandas: 1.4.0
   matplotlib: 3.5.3
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 32
         prefix: libopenblas
       filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: SkylakeX

       user_api: blas
   internal_api: openblas
    num_threads: 32
         prefix: libopenblas
       filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/scipy.libs/libopenblasp-r0-01191904.3.27.so
        version: 0.3.27
threading_layer: pthreads
   architecture: SkylakeX

       user_api: openmp
   internal_api: openmp
    num_threads: 32
         prefix: libgomp
       filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

COMMENT:
> Not really because my application is Object Detection, where AP (or mAP) is the main metric used by most papers and benchmarks (e.g. COCO: https://cocodataset.org/#detection-eval). In that case you can't predict all positives (or you would have an infinite number of false positives). That's what I meant by saying that when the model has 0 confidence in my example, it means it missed completely the detection, so it should never be counted as a positive no matter the threshold. Recall can't be above 0.55 in this case.

1. AP and mAP are not interchangeable, they are different.
2. MS COCO calculates things like `AP@IOU0.5`, `AP@IOU0.75` etc. These are completely different metrics compared to `average_precision_score`. And they define AP as `AP and AR are averaged over multiple Intersection over Union (IoU) values. Specifically we use 10 IoU thresholds of .50:.05:.95.`
3. MS COCO also use completely different interpolation method, while scikit-learn do not interpolate at all.

Labels: Bug
average_precision_score not working as expected

### Describe the bug

When compute AP with average_precision_score, I get unexpected results. The y_scores (output from the models) are very low for positive samples, so my AP should be very low. Instead I get a perfect 1.0 AP score.

### Steps/Code to Reproduce

```python
from sklearn.metrics import average_precision_score

# labels
y_true = [1, 1, 1, 1, 1]  # 5 positives
# Predicted scores
y_scores = [0.1, 0.3, 0.1, 0.2, 0.1]  # Model's confidence in predictions

# Calculate average precision
average_precision = average_precision_score(y_true, y_scores)
print("AP score:", average_precision)
```

### Expected Results

An AP score close to 0.

### Actual Results

AP score: 1.0

### Versions

```shell
System:
    python: 3.9.19 (main, May  6 2024, 19:43:03)  [GCC 11.2.0]
executable: /bin/python
   machine: Linux-5.15.0-121-generic-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.5.2
          pip: 24.2
   setuptools: 75.1.0
        numpy: 1.26.4
        scipy: 1.13.1
       Cython: None
       pandas: 1.4.0
   matplotlib: 3.5.3
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 32
         prefix: libopenblas
       filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: SkylakeX

       user_api: blas
   internal_api: openblas
    num_threads: 32
         prefix: libopenblas
       filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/scipy.libs/libopenblasp-r0-01191904.3.27.so
        version: 0.3.27
threading_layer: pthreads
   architecture: SkylakeX

       user_api: openmp
   internal_api: openmp
    num_threads: 32
         prefix: libgomp
       filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

COMMENT:
1. mAP is average across classes, but with only one class, mAP and AP should be equal.

2. `AP@IOU0.5` is Average Precision. @IOU0.5 simply means that IoU of 0.5 at least is needed to consider a prediction as matching an annotation. Once this matching is done, it should not change the way the AP is computed.

3. Interpolation is not what causes the major difference here. 

Even if you look at the formula given in the scikit-learn documentation: https://scikit-learn.org/1.5/modules/generated/sklearn.metrics.average_precision_score.html
What the function is doing right now is adding an artificial value of R_n = 1.0 that doesn't exist in the data. All I am saying is that this is not clear in the function documentation and it's not the behavior that is always expected when computing AP. Always reaching 100% Recall is not guaranteed in all applications of ML but this is what is currently assumed by  `average_precision_score`.

Labels: Bug
average_precision_score not working as expected

### Describe the bug

When compute AP with average_precision_score, I get unexpected results. The y_scores (output from the models) are very low for positive samples, so my AP should be very low. Instead I get a perfect 1.0 AP score.

### Steps/Code to Reproduce

```python
from sklearn.metrics import average_precision_score

# labels
y_true = [1, 1, 1, 1, 1]  # 5 positives
# Predicted scores
y_scores = [0.1, 0.3, 0.1, 0.2, 0.1]  # Model's confidence in predictions

# Calculate average precision
average_precision = average_precision_score(y_true, y_scores)
print("AP score:", average_precision)
```

### Expected Results

An AP score close to 0.

### Actual Results

AP score: 1.0

### Versions

```shell
System:
    python: 3.9.19 (main, May  6 2024, 19:43:03)  [GCC 11.2.0]
executable: /bin/python
   machine: Linux-5.15.0-121-generic-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.5.2
          pip: 24.2
   setuptools: 75.1.0
        numpy: 1.26.4
        scipy: 1.13.1
       Cython: None
       pandas: 1.4.0
   matplotlib: 3.5.3
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 32
         prefix: libopenblas
       filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: SkylakeX

       user_api: blas
   internal_api: openblas
    num_threads: 32
         prefix: libopenblas
       filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/scipy.libs/libopenblasp-r0-01191904.3.27.so
        version: 0.3.27
threading_layer: pthreads
   architecture: SkylakeX

       user_api: openmp
   internal_api: openmp
    num_threads: 32
         prefix: libgomp
       filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

COMMENT:
1. The point was that in MS COCO AP is the average between 10 different IOUs, which is not `average_precision_score`;
2. Interpolation is a huge factor since it is basically the only difference between `AUPRC`, `average_precision_score`, MS COCO AP, PASCAL VOC AP and so on. Different interpolations yield very different results.

scikit-learn does not add point for recall==1, it actually only adds (0, 1) point (precision = 1 and recall = 0)
https://github.com/scikit-learn/scikit-learn/blob/6e9039160f0dfc3153643143af4cfdca941d2045/sklearn/metrics/_ranking.py#L1040

If you remove this point you will get `average_precision_score` = `0.9110476661020139`

Can you please share the code that you used to draw the first graph, so that we can better understand the problem.

Labels: Bug
average_precision_score not working as expected

### Describe the bug

When compute AP with average_precision_score, I get unexpected results. The y_scores (output from the models) are very low for positive samples, so my AP should be very low. Instead I get a perfect 1.0 AP score.

### Steps/Code to Reproduce

```python
from sklearn.metrics import average_precision_score

# labels
y_true = [1, 1, 1, 1, 1]  # 5 positives
# Predicted scores
y_scores = [0.1, 0.3, 0.1, 0.2, 0.1]  # Model's confidence in predictions

# Calculate average precision
average_precision = average_precision_score(y_true, y_scores)
print("AP score:", average_precision)
```

### Expected Results

An AP score close to 0.

### Actual Results

AP score: 1.0

### Versions

```shell
System:
    python: 3.9.19 (main, May  6 2024, 19:43:03)  [GCC 11.2.0]
executable: /bin/python
   machine: Linux-5.15.0-121-generic-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.5.2
          pip: 24.2
   setuptools: 75.1.0
        numpy: 1.26.4
        scipy: 1.13.1
       Cython: None
       pandas: 1.4.0
   matplotlib: 3.5.3
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 32
         prefix: libopenblas
       filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: SkylakeX

       user_api: blas
   internal_api: openblas
    num_threads: 32
         prefix: libopenblas
       filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/scipy.libs/libopenblasp-r0-01191904.3.27.so
        version: 0.3.27
threading_layer: pthreads
   architecture: SkylakeX

       user_api: openmp
   internal_api: openmp
    num_threads: 32
         prefix: libgomp
       filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

COMMENT:
So if I replace this line 
https://github.com/scikit-learn/scikit-learn/blob/49c59480640e00a1b2bab18f0367ea0ff22b2c26/sklearn/metrics/_ranking.py#L849
with
`threshold_idxs = distinct_value_indices`
And this line
https://github.com/scikit-learn/scikit-learn/blob/49c59480640e00a1b2bab18f0367ea0ff22b2c26/sklearn/metrics/_ranking.py#L1033
with
`recall = tps / (y_true.size - 1)`

I will get `average_precision_score` = `0.5055340559052119`.

In terms of plots, we are basically will be doing this
![image](https://github.com/user-attachments/assets/b8507954-f2c6-4d9c-ace0-dfcc4cf61340)

instead of this
![image](https://github.com/user-attachments/assets/6ce8b9e9-a91b-40c6-9d67-af3f4e50b9d3)

@glemaitre can someone please tell what is the meaning of adding the last value to the tps? The only thing I could find was commentary `We also concatenate a value for the end of the curve.` and that's it.

Labels: Bug
average_precision_score not working as expected

### Describe the bug

When compute AP with average_precision_score, I get unexpected results. The y_scores (output from the models) are very low for positive samples, so my AP should be very low. Instead I get a perfect 1.0 AP score.

### Steps/Code to Reproduce

```python
from sklearn.metrics import average_precision_score

# labels
y_true = [1, 1, 1, 1, 1]  # 5 positives
# Predicted scores
y_scores = [0.1, 0.3, 0.1, 0.2, 0.1]  # Model's confidence in predictions

# Calculate average precision
average_precision = average_precision_score(y_true, y_scores)
print("AP score:", average_precision)
```

### Expected Results

An AP score close to 0.

### Actual Results

AP score: 1.0

### Versions

```shell
System:
    python: 3.9.19 (main, May  6 2024, 19:43:03)  [GCC 11.2.0]
executable: /bin/python
   machine: Linux-5.15.0-121-generic-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.5.2
          pip: 24.2
   setuptools: 75.1.0
        numpy: 1.26.4
        scipy: 1.13.1
       Cython: None
       pandas: 1.4.0
   matplotlib: 3.5.3
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 32
         prefix: libopenblas
       filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: SkylakeX

       user_api: blas
   internal_api: openblas
    num_threads: 32
         prefix: libopenblas
       filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/scipy.libs/libopenblasp-r0-01191904.3.27.so
        version: 0.3.27
threading_layer: pthreads
   architecture: SkylakeX

       user_api: openmp
   internal_api: openmp
    num_threads: 32
         prefix: libgomp
       filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

COMMENT:
> @glemaitre can someone please tell what is the meaning of adding the last value to the tps? The only thing I could find was commentary We also concatenate a value for the end of the curve. and that's it.

@glevv It corresponds to the prevalence. If you are in a medical setting, it would be the performance of a dummy classifier that always predict the positive outcome. Basically this is the chance level point. In this setting, you will never reach a precision of 0.0.

Labels: Bug
average_precision_score not working as expected

### Describe the bug

When compute AP with average_precision_score, I get unexpected results. The y_scores (output from the models) are very low for positive samples, so my AP should be very low. Instead I get a perfect 1.0 AP score.

### Steps/Code to Reproduce

```python
from sklearn.metrics import average_precision_score

# labels
y_true = [1, 1, 1, 1, 1]  # 5 positives
# Predicted scores
y_scores = [0.1, 0.3, 0.1, 0.2, 0.1]  # Model's confidence in predictions

# Calculate average precision
average_precision = average_precision_score(y_true, y_scores)
print("AP score:", average_precision)
```

### Expected Results

An AP score close to 0.

### Actual Results

AP score: 1.0

### Versions

```shell
System:
    python: 3.9.19 (main, May  6 2024, 19:43:03)  [GCC 11.2.0]
executable: /bin/python
   machine: Linux-5.15.0-121-generic-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.5.2
          pip: 24.2
   setuptools: 75.1.0
        numpy: 1.26.4
        scipy: 1.13.1
       Cython: None
       pandas: 1.4.0
   matplotlib: 3.5.3
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 32
         prefix: libopenblas
       filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: SkylakeX

       user_api: blas
   internal_api: openblas
    num_threads: 32
         prefix: libopenblas
       filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/scipy.libs/libopenblasp-r0-01191904.3.27.so
        version: 0.3.27
threading_layer: pthreads
   architecture: SkylakeX

       user_api: openmp
   internal_api: openmp
    num_threads: 32
         prefix: libgomp
       filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

COMMENT:
I have to put some thought if we should have a way to control this ending point depending on the application.

Labels: Bug
average_precision_score not working as expected

### Describe the bug

When compute AP with average_precision_score, I get unexpected results. The y_scores (output from the models) are very low for positive samples, so my AP should be very low. Instead I get a perfect 1.0 AP score.

### Steps/Code to Reproduce

```python
from sklearn.metrics import average_precision_score

# labels
y_true = [1, 1, 1, 1, 1]  # 5 positives
# Predicted scores
y_scores = [0.1, 0.3, 0.1, 0.2, 0.1]  # Model's confidence in predictions

# Calculate average precision
average_precision = average_precision_score(y_true, y_scores)
print("AP score:", average_precision)
```

### Expected Results

An AP score close to 0.

### Actual Results

AP score: 1.0

### Versions

```shell
System:
    python: 3.9.19 (main, May  6 2024, 19:43:03)  [GCC 11.2.0]
executable: /bin/python
   machine: Linux-5.15.0-121-generic-x86_64-with-glibc2.31

Python dependencies:
      sklearn: 1.5.2
          pip: 24.2
   setuptools: 75.1.0
        numpy: 1.26.4
        scipy: 1.13.1
       Cython: None
       pandas: 1.4.0
   matplotlib: 3.5.3
       joblib: 1.4.2
threadpoolctl: 3.5.0

Built with OpenMP: True

threadpoolctl info:
       user_api: blas
   internal_api: openblas
    num_threads: 32
         prefix: libopenblas
       filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/numpy.libs/libopenblas64_p-r0-0cf96a72.3.23.dev.so
        version: 0.3.23.dev
threading_layer: pthreads
   architecture: SkylakeX

       user_api: blas
   internal_api: openblas
    num_threads: 32
         prefix: libopenblas
       filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/scipy.libs/libopenblasp-r0-01191904.3.27.so
        version: 0.3.27
threading_layer: pthreads
   architecture: SkylakeX

       user_api: openmp
   internal_api: openmp
    num_threads: 32
         prefix: libgomp
       filepath: user/anaconda3/envs/perception39/lib/python3.9/site-packages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0
        version: None
```

COMMENT:
> This won't affect the calculation because a single point has zero area under curve.

It does because because the line between the last actual observed recall and this assumed recall of 1.0 has an area (see @glevv plots above).


I think in the end it is really a question of context. In pure classification, the current implementation make sense, but in other context it might not. 
TorchMetrics makes the difference by having an AP function specifically under its classification submodule and a mAP function under its detection submodule (with an option to get the result per class which is the AP).

Labels: Bug