Repository: scikit-learn/scikit-learn
Pull request: FIX Ignore zero sample weights in precision recall curve
Fix #16065.

Supersede and close #16319.

Problem statement:
[Feature Request/Improvement] Dealing with weight=0 in precision_recall_curve
It seems that when **sample_weight** is specified in **precision_recall_curve** function, all the scores in y_score will be reserved in the output thresholds, even though the corresponding weight of the score is zero.

```python
from sklearn.metrics import precision_recall_curve  
y_true1 = [0, 0, 1, 1, 1]
y_score1 = [0, 1, 0, 1, 2]
weight1 = [1, 1, 1, 2, 0]

y_true2 = [0, 0, 1, 1]
y_score2 = [0, 1, 0, 1]
weight2 = [1, 1, 1, 2]

precision1, recall1, thre1 = precision_recall_curve(y_true1, y_score1, sample_weight=weight1)
print('precisions1', precision1)
print('recalls1   ', recall1)
print('thresholds1', thre1)

precision2, recall2, thre2 = precision_recall_curve(y_true2, y_score2, sample_weight=weight2)
print('precisions2', precision2)
print('recalls2   ', recall2)
print('thresholds2', thre2)
```
Take the code above as an example. The weight assigned to score 2 is 0. I think the precision-recall curve got from input 1 and input 2 should be the same while found the results are 
```python
precisions1 [0.6        0.66666667 0.         1.        ]
recalls1    [1.         0.66666667 0.         0.        ]
thresholds1 [0 1 2]
precisions2 [0.6        0.66666667 1.        ]
recalls2    [1.         0.66666667 0.        ]
thresholds2 [0 1]
```
The precision-recall curve got from input 1 looked very strange for its sudden drop to 0.

Hints from issue discussion:
Maybe it's because the threshold of the input, they are very different because the first prediction value appears 2, which lead to 2 in threshold1`thresholds1 [0 1 2]`.
> Maybe it's because the threshold of the input, they are very different because the first prediction value appears 2, which lead to 2 in threshold1`thresholds1 [0 1 2]`.

Yes, I understand how it happens. It is not difficult to avoid this situation but it may be confusing sometimes. I just think prediction value 2 with weight 0 should not affects the curve.
> > Maybe it's because the threshold of the input, they are very different because the first prediction value appears 2, which lead to 2 in threshold1`thresholds1 [0 1 2]`.
> 
> Yes, I understand how it happens. It is not difficult to avoid this situation but it may be confusing sometimes. I just think prediction value 2 with weight 0 should not affects the curve.

Yes I agree with you. it seems more reasonable to filter examples with zero weight and then to decide the threshold using remaining examples.

I think @qinhanmin2014 or @thomasjpfan may be able to help here?

Commit message: Add whatsnew

Patch to review:
diff --git a/doc/whats_new/v0.24.rst b/doc/whats_new/v0.24.rst
index c3f3f993a2f57..72520c9b2a597 100644
--- a/doc/whats_new/v0.24.rst
+++ b/doc/whats_new/v0.24.rst
@@ -379,6 +379,12 @@ Changelog
   ``labels`` parameter.
   :pr:`17935` by :user:`Cary Goltermann <Ultramann>`.
 
+- |Fix| Samples with zero `sample_weight` values do not affect the results
+  from :func:`metrics.det_curve`, :func:`metrics.precision_recall_curve`
+  and :func:`metrics.roc_curve`.
+  :pr:`18328` by :user:`Albert Villanova del Moral <albertvillanova>` and
+  :user:`Alonso Silva Allende <alonsosilvaallende>`.
+
 :mod:`sklearn.model_selection`
 ..............................
 
diff --git a/sklearn/metrics/_ranking.py b/sklearn/metrics/_ranking.py
index 3813d3ed6e4d3..33b4db0d68903 100644
--- a/sklearn/metrics/_ranking.py
+++ b/sklearn/metrics/_ranking.py
@@ -27,6 +27,7 @@
 
 from ..utils import assert_all_finite
 from ..utils import check_consistent_length
+from ..utils.validation import _check_sample_weight
 from ..utils import column_or_1d, check_array
 from ..utils.multiclass import type_of_target
 from ..utils.extmath import stable_cumsum
@@ -283,14 +284,14 @@ def det_curve(y_true, y_score, pos_label=None, sample_weight=None):
     >>> thresholds
     array([0.35, 0.4 , 0.8 ])
     """
-    if len(np.unique(y_true)) != 2:
-        raise ValueError("Only one class present in y_true. Detection error "
-                         "tradeoff curve is not defined in that case.")
-
     fps, tps, thresholds = _binary_clf_curve(
         y_true, y_score, pos_label=pos_label, sample_weight=sample_weight
     )
 
+    if len(np.unique(y_true)) != 2:
+        raise ValueError("Only one class present in y_true. Detection error "
+                         "tradeoff curve is not defined in that case.")
+
     fns = tps[-1] - tps
     p_count = tps[-1]
     n_count = fps[-1]
@@ -688,8 +689,16 @@ def _binary_clf_curve(y_true, y_score, pos_label=None, sample_weight=None):
     assert_all_finite(y_true)
     assert_all_finite(y_score)
 
+    # Ensure non-negative sample_weight
     if sample_weight is not None:
         sample_weight = column_or_1d(sample_weight)
+        # Check to make sure sample_weight is non-negative and filter out
+        # zero-weighted samples, as they should not impact the score
+        _check_sample_weight(sample_weight, y_true, ensure_nonnegative=True)
+        nonzero_weight_mask = sample_weight > 0
+        y_true = y_true[nonzero_weight_mask]
+        y_score = y_score[nonzero_weight_mask]
+        sample_weight = sample_weight[nonzero_weight_mask]
 
     # ensure binary classification if pos_label is not specified
     # classes.dtype.kind in ('O', 'U', 'S') is required to avoid
@@ -742,7 +751,7 @@ def _binary_clf_curve(y_true, y_score, pos_label=None, sample_weight=None):
 
 
 @_deprecate_positional_args
-def precision_recall_curve(y_true, probas_pred, *, pos_label=None,
+def precision_recall_curve(y_true, y_score, *, pos_label=None,
                            sample_weight=None):
     """Compute precision-recall pairs for different probability thresholds.
 
@@ -769,8 +778,10 @@ def precision_recall_curve(y_true, probas_pred, *, pos_label=None,
         True binary labels. If labels are not either {-1, 1} or {0, 1}, then
         pos_label should be explicitly given.
 
-    probas_pred : ndarray of shape (n_samples,)
-        Estimated probabilities or decision function.
+    y_score : ndarray of shape (n_samples,)
+        Target scores, can either be probability estimates of the positive
+        class, or non-thresholded measure of decisions (as returned by
+        `decision_function` on some classifiers).
 
     pos_label : int or str, default=None
         The label of the positive class.
@@ -792,7 +803,7 @@ def precision_recall_curve(y_true, probas_pred, *, pos_label=None,
 
     thresholds : ndarray of shape (n_thresholds,)
         Increasing thresholds on the decision function used to compute
-        precision and recall. n_thresholds <= len(np.unique(probas_pred)).
+        precision and recall. n_thresholds <= len(np.unique(y_score)).
 
     See also
     --------
@@ -821,7 +832,8 @@ def precision_recall_curve(y_true, probas_pred, *, pos_label=None,
     array([0.35, 0.4 , 0.8 ])
 
     """
-    fps, tps, thresholds = _binary_clf_curve(y_true, probas_pred,
+
+    fps, tps, thresholds = _binary_clf_curve(y_true, y_score,
                                              pos_label=pos_label,
                                              sample_weight=sample_weight)
 
diff --git a/sklearn/metrics/tests/test_ranking.py b/sklearn/metrics/tests/test_ranking.py
index 166ff775e2690..c3629d89b8568 100644
--- a/sklearn/metrics/tests/test_ranking.py
+++ b/sklearn/metrics/tests/test_ranking.py
@@ -69,16 +69,16 @@ def make_prediction(dataset=None, binary=False):
 
     # run classifier, get class probabilities and label predictions
     clf = svm.SVC(kernel='linear', probability=True, random_state=0)
-    probas_pred = clf.fit(X[:half], y[:half]).predict_proba(X[half:])
+    y_score = clf.fit(X[:half], y[:half]).predict_proba(X[half:])
 
     if binary:
         # only interested in probabilities of the positive case
         # XXX: do we really want a special API for the binary case?
-        probas_pred = probas_pred[:, 1]
+        y_score = y_score[:, 1]
 
     y_pred = clf.predict(X[half:])
     y_true = y[half:]
-    return y_true, y_pred, probas_pred
+    return y_true, y_pred, y_score
 
 
 ###############################################################################
@@ -179,14 +179,14 @@ def _partial_roc(y_true, y_predict, max_fpr):
 @pytest.mark.parametrize('drop', [True, False])
 def test_roc_curve(drop):
     # Test Area under Receiver Operating Characteristic (ROC) curve
-    y_true, _, probas_pred = make_prediction(binary=True)
-    expected_auc = _auc(y_true, probas_pred)
+    y_true, _, y_score = make_prediction(binary=True)
+    expected_auc = _auc(y_true, y_score)
 
-    fpr, tpr, thresholds = roc_curve(y_true, probas_pred,
+    fpr, tpr, thresholds = roc_curve(y_true, y_score,
                                      drop_intermediate=drop)
     roc_auc = auc(fpr, tpr)
     assert_array_almost_equal(roc_auc, expected_auc, decimal=2)
-    assert_almost_equal(roc_auc, roc_auc_score(y_true, probas_pred))
+    assert_almost_equal(roc_auc, roc_auc_score(y_true, y_score))
     assert fpr.shape == tpr.shape
     assert fpr.shape == thresholds.shape
 
@@ -207,13 +207,13 @@ def test_roc_curve_end_points():
 def test_roc_returns_consistency():
     # Test whether the returned threshold matches up with tpr
     # make small toy dataset
-    y_true, _, probas_pred = make_prediction(binary=True)
-    fpr, tpr, thresholds = roc_curve(y_true, probas_pred)
+    y_true, _, y_score = make_prediction(binary=True)
+    fpr, tpr, thresholds = roc_curve(y_true, y_score)
 
     # use the given thresholds to determine the tpr
     tpr_correct = []
     for t in thresholds:
-        tp = np.sum((probas_pred >= t) & y_true)
+        tp = np.sum((y_score >= t) & y_true)
         p = np.sum(y_true)
         tpr_correct.append(1.0 * tp / p)
 
@@ -225,17 +225,17 @@ def test_roc_returns_consistency():
 
 def test_roc_curve_multi():
     # roc_curve not applicable for multi-class problems
-    y_true, _, probas_pred = make_prediction(binary=False)
+    y_true, _, y_score = make_prediction(binary=False)
 
     with pytest.raises(ValueError):
-        roc_curve(y_true, probas_pred)
+        roc_curve(y_true, y_score)
 
 
 def test_roc_curve_confidence():
     # roc_curve for confidence scores
-    y_true, _, probas_pred = make_prediction(binary=True)
+    y_true, _, y_score = make_prediction(binary=True)
 
-    fpr, tpr, thresholds = roc_curve(y_true, probas_pred - 0.5)
+    fpr, tpr, thresholds = roc_curve(y_true, y_score - 0.5)
     roc_auc = auc(fpr, tpr)
     assert_array_almost_equal(roc_auc, 0.90, decimal=2)
     assert fpr.shape == tpr.shape
@@ -244,7 +244,7 @@ def test_roc_curve_confidence():
 
 def test_roc_curve_hard():
     # roc_curve for hard decisions
-    y_true, pred, probas_pred = make_prediction(binary=True)
+    y_true, pred, y_score = make_prediction(binary=True)
 
     # always predict one
     trivial_pred = np.ones(y_true.shape)
@@ -664,20 +664,34 @@ def test_auc_score_non_binary_class():
             roc_auc_score(y_true, y_pred)
 
 
-def test_binary_clf_curve_multiclass_error():
+@pytest.mark.parametrize("curve_func", [
+    det_curve,
+    precision_recall_curve,
+    roc_curve,
+])
+def test_binary_clf_curve_multiclass_error(curve_func):
     rng = check_random_state(404)
     y_true = rng.randint(0, 3, size=10)
     y_pred = rng.rand(10)
     msg = "multiclass format is not supported"
-
     with pytest.raises(ValueError, match=msg):
-        precision_recall_curve(y_true, y_pred)
+        curve_func(y_true, y_pred)
+
 
+@pytest.mark.parametrize("curve_func", [
+    det_curve,
+    precision_recall_curve,
+    roc_curve,
+])
+def test_binary_clf_curve_nonnegative_sample_weight_error(curve_func):
+    # Contains negative sample weights
+    msg = "Sample weights must be non-negative"
     with pytest.raises(ValueError, match=msg):
-        roc_curve(y_true, y_pred)
+        curve_func([0, 1], [0, 1], sample_weight=[1, -1])
 
 
 @pytest.mark.parametrize("curve_func", [
+    det_curve,
     precision_recall_curve,
     roc_curve,
 ])
@@ -689,10 +703,10 @@ def test_binary_clf_curve_implicit_pos_label(curve_func):
            "value in {0, 1} or {-1, 1} or pass pos_label "
            "explicitly.")
     with pytest.raises(ValueError, match=msg):
-        roc_curve(np.array(["a", "b"], dtype='<U1'), [0., 1.])
+        curve_func(np.array(["a", "b"], dtype='<U1'), [0., 1.])
 
     with pytest.raises(ValueError, match=msg):
-        roc_curve(np.array(["a", "b"], dtype=object), [0., 1.])
+        curve_func(np.array(["a", "b"], dtype=object), [0., 1.])
 
     # The error message is slightly different for bytes-encoded
     # class labels, but otherwise the behavior is the same:
@@ -701,25 +715,43 @@ def test_binary_clf_curve_implicit_pos_label(curve_func):
            "value in {0, 1} or {-1, 1} or pass pos_label "
            "explicitly.")
     with pytest.raises(ValueError, match=msg):
-        roc_curve(np.array([b"a", b"b"], dtype='<S1'), [0., 1.])
+        curve_func(np.array([b"a", b"b"], dtype='<S1'), [0., 1.])
 
     # Check that it is possible to use floating point class labels
     # that are interpreted similarly to integer class labels:
     y_pred = [0., 1., 0.2, 0.42]
-    int_curve = roc_curve([0, 1, 1, 0], y_pred)
-    float_curve = roc_curve([0., 1., 1., 0.], y_pred)
+    int_curve = curve_func([0, 1, 1, 0], y_pred)
+    float_curve = curve_func([0., 1., 1., 0.], y_pred)
     for int_curve_part, float_curve_part in zip(int_curve, float_curve):
         np.testing.assert_allclose(int_curve_part, float_curve_part)
 
 
+@pytest.mark.parametrize("curve_func", [
+    det_curve,
+    precision_recall_curve,
+    roc_curve,
+])
+def test_binary_clf_curve_zero_sample_weight(curve_func):
+    y_true = [0, 0, 1, 1, 1]
+    y_score = [0.1, 0.2, 0.3, 0.4, 0.5]
+    sample_weight = [1, 1, 1, 0.5, 0]
+
+    result_1 = curve_func(y_true, y_score, sample_weight=sample_weight)
+    result_2 = curve_func(y_true[:-1], y_score[:-1],
+                          sample_weight=sample_weight[:-1])
+
+    for arr_1, arr_2 in zip(result_1, result_2):
+        assert_allclose(arr_1, arr_2)
+
+
 def test_precision_recall_curve():
-    y_true, _, probas_pred = make_prediction(binary=True)
-    _test_precision_recall_curve(y_true, probas_pred)
+    y_true, _, y_score = make_prediction(binary=True)
+    _test_precision_recall_curve(y_true, y_score)
 
     # Use {-1, 1} for labels; make sure original labels aren't modified
     y_true[np.where(y_true == 0)] = -1
     y_true_copy = y_true.copy()
-    _test_precision_recall_curve(y_true, probas_pred)
+    _test_precision_recall_curve(y_true, y_score)
     assert_array_equal(y_true_copy, y_true)
 
     labels = [1, 0, 0, 1]
@@ -732,31 +764,24 @@ def test_precision_recall_curve():
     assert p.size == t.size + 1
 
 
-def _test_precision_recall_curve(y_true, probas_pred):
+def _test_precision_recall_curve(y_true, y_score):
     # Test Precision-Recall and aread under PR curve
-    p, r, thresholds = precision_recall_curve(y_true, probas_pred)
-    precision_recall_auc = _average_precision_slow(y_true, probas_pred)
+    p, r, thresholds = precision_recall_curve(y_true, y_score)
+    precision_recall_auc = _average_precision_slow(y_true, y_score)
     assert_array_almost_equal(precision_recall_auc, 0.859, 3)
     assert_array_almost_equal(precision_recall_auc,
-                              average_precision_score(y_true, probas_pred))
+                              average_precision_score(y_true, y_score))
     # `_average_precision` is not very precise in case of 0.5 ties: be tolerant
-    assert_almost_equal(_average_precision(y_true, probas_pred),
+    assert_almost_equal(_average_precision(y_true, y_score),
                         precision_recall_auc, decimal=2)
     assert p.size == r.size
     assert p.size == thresholds.size + 1
     # Smoke test in the case of proba having only one value
-    p, r, thresholds = precision_recall_curve(y_true,
-                                              np.zeros_like(probas_pred))
+    p, r, thresholds = precision_recall_curve(y_true, np.zeros_like(y_score))
     assert p.size == r.size
     assert p.size == thresholds.size + 1
 
 
-def test_precision_recall_curve_errors():
-    # Contains non-binary labels
-    with pytest.raises(ValueError):
-        precision_recall_curve([0, 1, 2], [[0.0], [1.0], [1.0]])
-
-
 def test_precision_recall_curve_toydata():
     with np.errstate(all="raise"):
         # Binary classification
@@ -908,20 +933,20 @@ def test_score_scale_invariance():
     # This test was expanded (added scaled_down) in response to github
     # issue #3864 (and others), where overly aggressive rounding was causing
     # problems for users with very small y_score values
-    y_true, _, probas_pred = make_prediction(binary=True)
+    y_true, _, y_score = make_prediction(binary=True)
 
-    roc_auc = roc_auc_score(y_true, probas_pred)
-    roc_auc_scaled_up = roc_auc_score(y_true, 100 * probas_pred)
-    roc_auc_scaled_down = roc_auc_score(y_true, 1e-6 * probas_pred)
-    roc_auc_shifted = roc_auc_score(y_true, probas_pred - 10)
+    roc_auc = roc_auc_score(y_true, y_score)
+    roc_auc_scaled_up = roc_auc_score(y_true, 100 * y_score)
+    roc_auc_scaled_down = roc_auc_score(y_true, 1e-6 * y_score)
+    roc_auc_shifted = roc_auc_score(y_true, y_score - 10)
     assert roc_auc == roc_auc_scaled_up
     assert roc_auc == roc_auc_scaled_down
     assert roc_auc == roc_auc_shifted
 
-    pr_auc = average_precision_score(y_true, probas_pred)
-    pr_auc_scaled_up = average_precision_score(y_true, 100 * probas_pred)
-    pr_auc_scaled_down = average_precision_score(y_true, 1e-6 * probas_pred)
-    pr_auc_shifted = average_precision_score(y_true, probas_pred - 10)
+    pr_auc = average_precision_score(y_true, y_score)
+    pr_auc_scaled_up = average_precision_score(y_true, 100 * y_score)
+    pr_auc_scaled_down = average_precision_score(y_true, 1e-6 * y_score)
+    pr_auc_shifted = average_precision_score(y_true, y_score - 10)
     assert pr_auc == pr_auc_scaled_up
     assert pr_auc == pr_auc_scaled_down
     assert pr_auc == pr_auc_shifted
@@ -949,8 +974,7 @@ def test_score_scale_invariance():
     ([1, 0, 1], [0.5, 0.75, 1], [1, 1, 0], [0, 0.5, 0.5]),
     ([1, 0, 1], [0.25, 0.5, 0.75], [1, 1, 0], [0, 0.5, 0.5]),
 ])
-def test_det_curve_toydata(y_true, y_score,
-                                                expected_fpr, expected_fnr):
+def test_det_curve_toydata(y_true, y_score, expected_fpr, expected_fnr):
     # Check on a batch of small examples.
     fpr, fnr, _ = det_curve(y_true, y_score)
 
diff --git a/sklearn/utils/validation.py b/sklearn/utils/validation.py
index a0c88ef36e281..79adb7ea6d5b9 100644
--- a/sklearn/utils/validation.py
+++ b/sklearn/utils/validation.py
@@ -1264,7 +1264,8 @@ def _check_psd_eigenvalues(lambdas, enable_warnings=False):
     return lambdas
 
 
-def _check_sample_weight(sample_weight, X, dtype=None):
+def _check_sample_weight(sample_weight, X, dtype=None,
+                         ensure_nonnegative=False):
     """Validate sample weights.
 
     Note that passing sample_weight=None will output an array of ones.
@@ -1280,13 +1281,17 @@ def _check_sample_weight(sample_weight, X, dtype=None):
     X : {ndarray, list, sparse matrix}
         Input data.
 
-    dtype: dtype
+    dtype : dtype
        dtype of the validated `sample_weight`.
        If None, and the input `sample_weight` is an array, the dtype of the
        input is preserved; otherwise an array with the default numpy dtype
        is be allocated.  If `dtype` is not one of `float32`, `float64`,
        `None`, the output will be of dtype `float64`.
 
+    ensure_nonnegative : bool, default=False
+        Check that `sample_weight` is non-negative, otherwise raise a value
+        error.
+
     Returns
     -------
     sample_weight : ndarray of shape (n_samples,)
@@ -1314,6 +1319,11 @@ def _check_sample_weight(sample_weight, X, dtype=None):
         if sample_weight.shape != (n_samples,):
             raise ValueError("sample_weight.shape == {}, expected {}!"
                              .format(sample_weight.shape, (n_samples,)))
+
+    if sample_weight is not None and ensure_nonnegative:
+        if not np.all(sample_weight >= 0):
+            raise ValueError("Sample weights must be non-negative")
+
     return sample_weight
 
 


Code review comments:
diff_hunk: @@ -742,7 +751,7 @@ def _binary_clf_curve(y_true, y_score, pos_label=None, sample_weight=None):
 
 
 @_deprecate_positional_args
-def precision_recall_curve(y_true, probas_pred, *, pos_label=None,
+def precision_recall_curve(y_true, y_score, *, pos_label=None,
path: sklearn/metrics/_ranking.py
text: @user1:
I agree `y_score` might have been better but we can't easily do this as it would break backward compatibility for people using key word arguments ( `precision_recall_curve(y_true=.., probas_pred=...)`.

If we do rename, we would probably need to go through a deprecation warning cycle. i.e. 
 - change `probas_pred` -> `y_score` or `y_pred` and make it an optional parameter.
 - add probas_pred as a key word argument.
 - raise a DeprecationWarning if probas_pred  is used, and make sure it's never used at the same time with y_pred

Let's wait for a second opinion, I'm not sure if it's worth to do it..

@author:
You are right. I did not think about people using keyword arguments for positional ones...

Let's wait for some other opinion to decide whether revert this or start a deprecation warning cycle.

@user2:
If we do not rename, this PR might be much easier to get merged.

@author:
I have reverted the renaming.

diff_hunk: @@ -1314,6 +1319,11 @@ def _check_sample_weight(sample_weight, X, dtype=None):
         if sample_weight.shape != (n_samples,):
             raise ValueError("sample_weight.shape == {}, expected {}!"
                              .format(sample_weight.shape, (n_samples,)))
+
+    if sample_weight is not None and ensure_nonnegative:
+        if not np.all(sample_weight >= 0):
+            raise ValueError("Sample weights must be non-negative")
path: sklearn/utils/validation.py
text: @user1:
Checking for negative sample weight is a separate issue (see discussion in https://github.com/scikit-learn/scikit-learn/issues/15531 and linked PRs). There are cases where negative sample weight make sense apparently..  So we should not do this check as part of this PR.

@author:
This code was already implemented by the superseded PR. I can remove it.