Repository: scikit-learn/scikit-learn
Pull request: FIX make `roc_auc_score` consistent with `roc_curve` with a single class
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/main/CONTRIBUTING.md
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->
Fixes https://github.com/scikit-learn/scikit-learn/issues/24381
closes #26176

See also #27047 

#### What does this implement/fix? Explain your changes.
 - replaced ValueError with UndefinedMetricWarning
 - replaced -0.0 with 0.0
 - updated tests

#### Any other comments?


<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->


Problem statement:
Inconsistency in AUC ROC and AUPR API
### Describe the bug

When only one class is present on the groundtruth. The function `roc_auc_score` throws an `ValueError` and exits while the `average_precision_score` returns `-0.0`. I feel that both functions should return similar output in this case.

### Steps/Code to Reproduce

```python
from sklearn.metrics import roc_auc_score,  average_precision_score

y_true = [1]*10
prob = [0.49]*10

auc_roc = roc_auc_score(y_true, prob)
aupr= average_precision_score(y_true, prob)

```

### Expected Results

auc_roc = roc_auc_score(y_true, prob) >> throws `ValueError: Only one class present in y_true. ROC AUC score is not defined in that case`

aupr=-0.0

### Actual Results

Both should have similar outputs

### Versions

```shell
Python dependencies:                                                                                                
      sklearn: 1.1.2                                                                                                
          pip: 22.0.2                                                                                               
   setuptools: 59.6.0                                                                                               
        numpy: 1.23.2                                                                                               
        scipy: 1.9.1                                                                                                
       Cython: None                                                                                                 
       pandas: 1.4.3                                                                                                
   matplotlib: 3.5.3                                                                                                
       joblib: 1.1.0                                                                                                
threadpoolctl: 3.1.0  

Built with OpenMP: True                                                                                             
                                                                                                                    
threadpoolctl info:          
       user_api: openmp      
   internal_api: openmp      
         prefix: libgomp     
       filepath: /home/abhijeet/Code/GMML/.gmml/lib/python3.10/sitepackages/scikit_learn.libs/libgomp-a34b3233.so.1.0.0    

version: None        
    num_threads: 24          

       user_api: blas        
   internal_api: openblas    
         prefix: libopenblas                              
       filepath: /home/abhijeet/Code/GMML/.gmml/lib/python3.10/site-packages/numpy.libs/libopenblas64_p-r0-742d56dc.
3.20.so                      
        version: 0.3.20      
threading_layer: pthreads    
   architecture: SkylakeX    
    num_threads: 24          

       user_api: blas        
   internal_api: openblas    
         prefix: libopenblas                              
       filepath: /home/abhijeet/Code/GMML/.gmml/lib/python3.10/site-packages/scipy.libs/libopenblasp-r0-9f9f5dbc.3.1
8.so                         
        version: 0.3.18      
threading_layer: pthreads    
   architecture: SkylakeX    
    num_threads: 24          

       user_api: openmp      
   internal_api: openmp      
         prefix: libgomp     
       filepath: /home/abhijeet/Code/GMML/.gmml/lib/python3.10/site-packages/torch/lib/libgomp-a34b3233.so.1        
        version: None        
    num_threads: 12
```

Hints from issue discussion:
`average_precision_score` is not an area under precision-recall curve, it's a different metric
@GLevV isn't it an discrete estimation of the area under the PR curve?
https://glassboxmedicine.com/2019/03/02/measuring-performance-auprc/

EDIT- added the correct link
Actually, they are different. 
Check [sklearn.metrics.precision_recall_curve](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_curve.html#) for PRcurve
The documentation here confirms that it is a particular kind of estimation of the area under the PR curve (or rather, that the PR curve itself needs to be interpolated between discrete points):

> AP summarizes a precision-recall curve [...]  This implementation is not interpolated and is different from computing the area under the precision-recall curve with the trapezoidal rule, which uses linear interpolation and can be too optimistic.

See also the Notes from the `PrecisionRecallDisplay`:

> The average precision (cf. `average_precision`) in scikit-learn is computed without any interpolation. To be consistent with this metric, the precision-recall curve is plotted without any interpolation as well (step-wise style).

---

As to the behavior, I get 1.0 instead of 0.0 when running your code.  If I make the only class something other than 1 (the default positive class), then I get `NaN` instead.  When the only class is the positive class, I think this is correct: at every threshold (other than the one making no positive predictions, where we just default precision to 1.0) the recall is well-defined and between 0 and 1 while precision is well-defined and exactly 1.  When instead the only class is the negative class, recall has a zero denominator.  I could see making the score raise an error instead of returning `NaN` in this case.  While it seems technically correct to call the score 1 when the only class is the positive one, I can agree that it could lead to misleading results if being aggregated in a cross-validation or such.

So I would not consider it a bug since there are no real "wrong" results.
We could improve the consistency since that now:

- Calling `roc_curve` and `ROCCurveDisplay` will not raise an error but a warning
- `roc_auc_score` will raise an error
- `precision_recall_curve`, `average_precision_score` and display will raise a warning.

Maybe having `roc_auc_score` raising a warning and having the same behaviour as the curve would be OK.
@glemaitre could I contribute a PR to improve the consistency of the API.

TODO would be :

- [ ] `roc_auc_score` should raise `Warning` instead of `Error`

Does that look okay?
I would like to have some thoughts other core-devs before going forward.
ping @Micky774 @thomasjpfan @adrinjalali @ogrisel 
I don't think we should change the behavior of `roc_auc_score`. https://github.com/scikit-learn/scikit-learn/issues/24381#issuecomment-1244075113 is about the PR curve, not the ROC curve.

For the record, here is what I observe (as @bmreiniger reported):

```python
>>> from sklearn.metrics import roc_auc_score,  average_precision_score
... average_precision_score([0] * 5, [0.4] * 5)
/Users/ogrisel/code/scikit-learn/sklearn/metrics/_ranking.py:891: UserWarning: No positive class found in y_true, recall is set to one for all thresholds.
  warnings.warn(
-0.0
>>> average_precision_score([0] * 5, [0.6] * 5)
/Users/ogrisel/code/scikit-learn/sklearn/metrics/_ranking.py:891: UserWarning: No positive class found in y_true, recall is set to one for all thresholds.
  warnings.warn(
-0.0
>>> average_precision_score([1] * 5, [0.4] * 5)
1.0
>>> average_precision_score([1] * 5, [0.6] * 5)
1.0
```

We could at least avoid returning a negative 0 which is confusing and instead report a positive zero.

I don't know if we should upgrade the warning to a `ValueError` or not.
Oh this again. I think I'd be happy with all the methods which are not raising, to warn in case one class is present.
Indeed it seems like two reasonable changes as presented are:
1. Update `roc_auc_score` to issue a warning instead of an error, and behave the same as `roc_curve`.
2. Return positive zero (`0`) instead of negative zero (`-0.0`).

I think both make sense here.

Commit message: bugfix

Patch to review:
diff --git a/doc/whats_new/v1.6.rst b/doc/whats_new/v1.6.rst
index e93eb01d3d32d..629944b7e52be 100644
--- a/doc/whats_new/v1.6.rst
+++ b/doc/whats_new/v1.6.rst
@@ -326,6 +326,10 @@ Changelog
   classification labels.
   :pr:`29738` by `Adrin Jalali`_.
 
+- |Fix| :func:`metrics.roc_auc_score` will now correctly return 0.0 and
+  warn user if only one class is present in the labels.
+  :pr:`27412` by :user:`Gleb Levitski <glevv>`.
+
 - |API| scoring="neg_max_error" should be used instead of
   scoring="max_error" which is now deprecated.
   :pr:`29462` by :user:`Farid "Freddie" Taba <artificialfintelligence>`.
diff --git a/sklearn/metrics/_ranking.py b/sklearn/metrics/_ranking.py
index fd5c30805a0c0..0d24a68bf464b 100644
--- a/sklearn/metrics/_ranking.py
+++ b/sklearn/metrics/_ranking.py
@@ -224,8 +224,9 @@ def _binary_uninterpolated_average_precision(
         )
         # Return the step function integral
         # The following works because the last entry of precision is
-        # guaranteed to be 1, as returned by precision_recall_curve
-        return -np.sum(np.diff(recall) * np.array(precision)[:-1])
+        # guaranteed to be 1, as returned by precision_recall_curve.
+        # Due to numerical error, we can get `-0.0` and we therefore clip it.
+        return max(0.0, -np.sum(np.diff(recall) * np.array(precision)[:-1]))
 
     y_type = type_of_target(y_true, input_name="y_true")
 
@@ -346,7 +347,7 @@ def det_curve(y_true, y_score, pos_label=None, sample_weight=None):
 
     if len(np.unique(y_true)) != 2:
         raise ValueError(
-            "Only one class present in y_true. Detection error "
+            "Only one class is present in y_true. Detection error "
             "tradeoff curve is not defined in that case."
         )
 
@@ -371,10 +372,15 @@ def det_curve(y_true, y_score, pos_label=None, sample_weight=None):
 def _binary_roc_auc_score(y_true, y_score, sample_weight=None, max_fpr=None):
     """Binary roc auc score."""
     if len(np.unique(y_true)) != 2:
-        raise ValueError(
-            "Only one class present in y_true. ROC AUC score "
-            "is not defined in that case."
+        warnings.warn(
+            (
+                "Only one class is present in y_true. ROC AUC score "
+                "is not defined in that case. The score is set to "
+                "0.0."
+            ),
+            UndefinedMetricWarning,
         )
+        return 0.0
 
     fpr, tpr, _ = roc_curve(y_true, y_score, sample_weight=sample_weight)
     if max_fpr is None or max_fpr == 1:
diff --git a/sklearn/metrics/tests/test_common.py b/sklearn/metrics/tests/test_common.py
index 7f9b4ee74f0f7..f70f0bfa50137 100644
--- a/sklearn/metrics/tests/test_common.py
+++ b/sklearn/metrics/tests/test_common.py
@@ -7,6 +7,7 @@
 
 from sklearn._config import config_context
 from sklearn.datasets import make_multilabel_classification
+from sklearn.exceptions import UndefinedMetricWarning
 from sklearn.metrics import (
     accuracy_score,
     average_precision_score,
@@ -840,8 +841,14 @@ def test_format_invariance_with_1d_vectors(name):
         if name not in (
             MULTIOUTPUT_METRICS | THRESHOLDED_MULTILABEL_METRICS | MULTILABELS_METRICS
         ):
-            with pytest.raises(ValueError):
-                metric(y1_row, y2_row)
+            if "roc_auc" in name:
+                # for consistency between the `roc_cuve` and `roc_auc_score`
+                # 0.0 is returned and an `UndefinedMetricWarning` is raised
+                with pytest.warns(UndefinedMetricWarning):
+                    assert metric(y1_row, y2_row) == pytest.approx(0.0)
+            else:
+                with pytest.raises(ValueError):
+                    metric(y1_row, y2_row)
 
 
 @pytest.mark.parametrize(
diff --git a/sklearn/metrics/tests/test_ranking.py b/sklearn/metrics/tests/test_ranking.py
index 87ee625f949e8..cfc2b71c4fabb 100644
--- a/sklearn/metrics/tests/test_ranking.py
+++ b/sklearn/metrics/tests/test_ranking.py
@@ -1,5 +1,4 @@
 import re
-import warnings
 
 import numpy as np
 import pytest
@@ -355,6 +354,24 @@ def test_roc_curve_toydata():
     assert_array_almost_equal(fpr, [0, 1])
     assert_almost_equal(roc_auc, 0.5)
 
+    y_true = [1, 1]
+    y_score = [0.25, 0.75]
+    # assert UndefinedMetricWarning because of no negative sample in y_true
+    expected_message = (
+        "No negative samples in y_true, false positive value should be meaningless"
+    )
+    with pytest.warns(UndefinedMetricWarning, match=expected_message):
+        tpr, fpr, _ = roc_curve(y_true, y_score)
+    assert_array_almost_equal(tpr, [np.nan, np.nan, np.nan])
+    assert_array_almost_equal(fpr, [0.0, 0.5, 1.0])
+    expected_message = (
+        "Only one class is present in y_true. "
+        "ROC AUC score is not defined in that case."
+    )
+    # should raise a warning since only one class is present
+    with pytest.warns(UndefinedMetricWarning, match=expected_message):
+        roc_auc_score(y_true, y_score)
+
     y_true = [0, 0]
     y_score = [0.25, 0.75]
     # assert UndefinedMetricWarning because of no positive sample in y_true
@@ -363,41 +380,31 @@ def test_roc_curve_toydata():
     )
     with pytest.warns(UndefinedMetricWarning, match=expected_message):
         tpr, fpr, _ = roc_curve(y_true, y_score)
-
-    with pytest.raises(ValueError):
-        roc_auc_score(y_true, y_score)
     assert_array_almost_equal(tpr, [0.0, 0.5, 1.0])
     assert_array_almost_equal(fpr, [np.nan, np.nan, np.nan])
-
-    y_true = [1, 1]
-    y_score = [0.25, 0.75]
-    # assert UndefinedMetricWarning because of no negative sample in y_true
     expected_message = (
-        "No negative samples in y_true, false positive value should be meaningless"
+        "Only one class is present in y_true. "
+        "ROC AUC score is not defined in that case."
     )
+    # should raise a warning since only one class is present
     with pytest.warns(UndefinedMetricWarning, match=expected_message):
-        tpr, fpr, _ = roc_curve(y_true, y_score)
-
-    with pytest.raises(ValueError):
         roc_auc_score(y_true, y_score)
-    assert_array_almost_equal(tpr, [np.nan, np.nan, np.nan])
-    assert_array_almost_equal(fpr, [0.0, 0.5, 1.0])
 
     # Multi-label classification task
     y_true = np.array([[0, 1], [0, 1]])
     y_score = np.array([[0, 1], [0, 1]])
-    with pytest.raises(ValueError):
+    with pytest.warns(UndefinedMetricWarning, match=expected_message):
         roc_auc_score(y_true, y_score, average="macro")
-    with pytest.raises(ValueError):
+    with pytest.warns(UndefinedMetricWarning, match=expected_message):
         roc_auc_score(y_true, y_score, average="weighted")
     assert_almost_equal(roc_auc_score(y_true, y_score, average="samples"), 1.0)
     assert_almost_equal(roc_auc_score(y_true, y_score, average="micro"), 1.0)
 
     y_true = np.array([[0, 1], [0, 1]])
     y_score = np.array([[0, 1], [1, 0]])
-    with pytest.raises(ValueError):
+    with pytest.warns(UndefinedMetricWarning, match=expected_message):
         roc_auc_score(y_true, y_score, average="macro")
-    with pytest.raises(ValueError):
+    with pytest.warns(UndefinedMetricWarning, match=expected_message):
         roc_auc_score(y_true, y_score, average="weighted")
     assert_almost_equal(roc_auc_score(y_true, y_score, average="samples"), 0.5)
     assert_almost_equal(roc_auc_score(y_true, y_score, average="micro"), 0.5)
@@ -814,30 +821,19 @@ def test_auc_score_non_binary_class():
     y_pred = rng.rand(10)
     # y_true contains only one class value
     y_true = np.zeros(10, dtype="int")
-    err_msg = "ROC AUC score is not defined"
-    with pytest.raises(ValueError, match=err_msg):
+    warn_message = (
+        "Only one class is present in y_true. "
+        "ROC AUC score is not defined in that case."
+    )
+    with pytest.warns(UndefinedMetricWarning, match=warn_message):
         roc_auc_score(y_true, y_pred)
     y_true = np.ones(10, dtype="int")
-    with pytest.raises(ValueError, match=err_msg):
+    with pytest.warns(UndefinedMetricWarning, match=warn_message):
         roc_auc_score(y_true, y_pred)
     y_true = np.full(10, -1, dtype="int")
-    with pytest.raises(ValueError, match=err_msg):
+    with pytest.warns(UndefinedMetricWarning, match=warn_message):
         roc_auc_score(y_true, y_pred)
 
-    with warnings.catch_warnings(record=True):
-        rng = check_random_state(404)
-        y_pred = rng.rand(10)
-        # y_true contains only one class value
-        y_true = np.zeros(10, dtype="int")
-        with pytest.raises(ValueError, match=err_msg):
-            roc_auc_score(y_true, y_pred)
-        y_true = np.ones(10, dtype="int")
-        with pytest.raises(ValueError, match=err_msg):
-            roc_auc_score(y_true, y_pred)
-        y_true = np.full(10, -1, dtype="int")
-        with pytest.raises(ValueError, match=err_msg):
-            roc_auc_score(y_true, y_pred)
-
 
 @pytest.mark.parametrize("curve_func", CURVE_FUNCS)
 def test_binary_clf_curve_multiclass_error(curve_func):
@@ -1332,8 +1328,8 @@ def test_det_curve_perfect_scores(y_true):
     [
         ([0, 1], [0, 0.5, 1], "inconsistent numbers of samples"),
         ([0, 1, 1], [0, 0.5], "inconsistent numbers of samples"),
-        ([0, 0, 0], [0, 0.5, 1], "Only one class present in y_true"),
-        ([1, 1, 1], [0, 0.5, 1], "Only one class present in y_true"),
+        ([0, 0, 0], [0, 0.5, 1], "Only one class is present in y_true"),
+        ([1, 1, 1], [0, 0.5, 1], "Only one class is present in y_true"),
         (
             ["cancer", "cancer", "not cancer"],
             [0.2, 0.3, 0.8],


Code review comments:
diff_hunk: @@ -355,6 +354,24 @@ def test_roc_curve_toydata():
     assert_array_almost_equal(fpr, [0, 1])
     assert_almost_equal(roc_auc, 0.5)
 
+    y_true = [1, 1]
path: sklearn/metrics/tests/test_ranking.py
text: @user1:
Do we really need to change the order here? I think in main this was below the case where y_true = [0, 0] and y_score = [0.25, 0.75]. So this difference seems extra.