Repository: scikit-learn/scikit-learn
Pull request: [MRG] FIX Correct brier_score_loss when there's only one class in y_true
closes #8459, closes #9300, closes #9301, closes #9562, closes #11245
Correct brier_score_loss when there's only one class in y_true.
This is an ugly solution, but will fix existing bugs, and avoid backward incompatibility.
Remove some redundant code in calibration_curve and brier_score_loss.

Problem statement:
brier_score_loss returns incorrect value when all y_true values are True/1
```python
In [7]: brier_score_loss(np.array([1, 1, 1]), np.array([1, 1, 1]))
Out[7]: 1.0

In [8]: brier_score_loss(np.array([0, 0, 0]), np.array([0, 0, 0]))
Out[8]: 0.0

In [9]: brier_score_loss(np.array([True, True, True]), np.array([1, 1, 1]))
Out[9]: 1.0

In [10]: brier_score_loss(np.array(["foo", "foo", "foo"]), np.array([1, 1, 1]), pos_label="foo")
Out[10]: 1.0
```
In all of these cases the output should be 0, as the `y_pred` correctly predicts `y_true`.

The function calls are `brier_score_loss` -> `_check_binary_probabilistic_predictions` -> `label_binarize`, with the issue starting in `label_binarize`.

`brier_score_loss` has a `pos_label` parameter, but setting this to 1, does not fix the issue. It then overwrites `y_true` with True and False based on the `pos_label` parameter. It then calls `_check_binary_probabilistic_predictions` to check the `y_true` and `y_pred` values, and returns the output of `label_binarize(y_true, labels)`, where labels are the unique values in `y_label`. Manually setting this to `[0, 1]` for the case where y_true is all 1s does not fix the issue.

In `label_binarize`, if there is only one class, the array is returned as 0 + the *negative* label

```python
    if y_type == "binary":
        if n_classes == 1:
            if sparse_output:
                return sp.csr_matrix((n_samples, 1), dtype=int)
            else:
                Y = np.zeros((len(y), 1), dtype=np.int)
                Y += neg_label
                return Y
```

The resulting comparison in `brier_score_loss` - `np.average((y_true - y_prob) ** 2, weights=sample_weight)` then has `y_true` values of all 0s instead of 1s, hence the incorrect score.


Operating system: Ubuntu 17.04 64 bit, problem present in master, 18.1 and 18.2.

brier_score_loss error
Hello. I think I found a mistake in brier_score_loss. 
When you have a target = 1 and a prediction = 1 too, brier_score_loss should be 0 (best result), but it gives 1. Why is it happening? Because _check_binary_probabilistic_predictions gets target with only one class and convert it to 0. And metric calculates for target = 0 and prediction = 1. The same problem for target = 1 and prediction = 0. brier_score_loss is 0 (the best result), but it should be 1. 

Examples:
Approx = [0, 0, 0, 0]
Target = [1, 1, 1, 1]
Weight = [1, 1, 1, 1]
brier_score_loss(Target, Approx, sample_weight=Weight) 
result is 0

Approx = [1, 1, 1, 1]
Target = [1, 1, 1, 1]
Weight = [1, 1, 1, 1]
brier_score_loss(Target, Approx, sample_weight=Weight) 
result is 1

Maybe we should fix it? Thank you.

Hints from issue discussion:
I have made a tentative pull request [here](https://github.com/scikit-learn/scikit-learn/pull/9301).
I'm still experiencing this issue in `0.19.1`. PR #9980 seems to address this, but #9300 and #9980 do not reference each other. This comment should fix that. 
Thanks. Duplicate issue of #9300, #8459
Sorry, just amended those links. #9301 is likely to be merged for next release
Thank you

Commit message: DOC parameter description

Patch to review:
diff --git a/doc/whats_new/v0.21.rst b/doc/whats_new/v0.21.rst
index 3175fca4747f6..5a23c452dcd42 100644
--- a/doc/whats_new/v0.21.rst
+++ b/doc/whats_new/v0.21.rst
@@ -397,6 +397,10 @@ Support for Python 3.4 and below has been officially dropped.
   and now it returns NaN and raises :class:`exceptions.UndefinedMetricWarning`.
   :issue:`12855` by :user:`Pawel Sendyk <psendyk>`.
 
+- |Fix| Fixed a bug where :func:`metrics.brier_score_loss` will sometimes
+  return incorrect result when there's only one class in ``y_true``.
+  :issue:`13628` by :user:`Hanmin Qin <qinhanmin2014>`.
+
 - |API| The parameter ``labels`` in :func:`metrics.hamming_loss` is deprecated
   in version 0.21 and will be removed in version 0.23.
   :issue:`10580` by :user:`Reshama Shaikh <reshamas>` and :user:`Sandra
diff --git a/sklearn/calibration.py b/sklearn/calibration.py
index 4329eba81a883..d3e09c25b6653 100644
--- a/sklearn/calibration.py
+++ b/sklearn/calibration.py
@@ -25,7 +25,6 @@
 from .isotonic import IsotonicRegression
 from .svm import LinearSVC
 from .model_selection import check_cv
-from .metrics.classification import _check_binary_probabilistic_predictions
 
 
 class CalibratedClassifierCV(BaseEstimator, ClassifierMixin):
@@ -572,6 +571,7 @@ def calibration_curve(y_true, y_prob, normalize=False, n_bins=5,
     """
     y_true = column_or_1d(y_true)
     y_prob = column_or_1d(y_prob)
+    check_consistent_length(y_true, y_prob)
 
     if normalize:  # Normalize predicted values into interval [0, 1]
         y_prob = (y_prob - y_prob.min()) / (y_prob.max() - y_prob.min())
@@ -579,7 +579,11 @@ def calibration_curve(y_true, y_prob, normalize=False, n_bins=5,
         raise ValueError("y_prob has values outside [0, 1] and normalize is "
                          "set to False.")
 
-    y_true = _check_binary_probabilistic_predictions(y_true, y_prob)
+    labels = np.unique(y_true)
+    if len(labels) > 2:
+        raise ValueError("Only binary classification is supported. "
+                         "Provided labels %s." % labels)
+    y_true = label_binarize(y_true, labels)[:, 0]
 
     if strategy == 'quantile':  # Determine bin edges by distribution of data
         quantiles = np.linspace(0, 1, n_bins + 1)
diff --git a/sklearn/metrics/classification.py b/sklearn/metrics/classification.py
index fad416a7fc0ff..475544ad64197 100644
--- a/sklearn/metrics/classification.py
+++ b/sklearn/metrics/classification.py
@@ -28,7 +28,7 @@
 from scipy.sparse import coo_matrix
 from scipy.sparse import csr_matrix
 
-from ..preprocessing import LabelBinarizer, label_binarize
+from ..preprocessing import LabelBinarizer
 from ..preprocessing import LabelEncoder
 from ..utils import assert_all_finite
 from ..utils import check_array
@@ -2301,25 +2301,6 @@ def hinge_loss(y_true, pred_decision, labels=None, sample_weight=None):
     return np.average(losses, weights=sample_weight)
 
 
-def _check_binary_probabilistic_predictions(y_true, y_prob):
-    """Check that y_true is binary and y_prob contains valid probabilities"""
-    check_consistent_length(y_true, y_prob)
-
-    labels = np.unique(y_true)
-
-    if len(labels) > 2:
-        raise ValueError("Only binary classification is supported. "
-                         "Provided labels %s." % labels)
-
-    if y_prob.max() > 1:
-        raise ValueError("y_prob contains values greater than 1.")
-
-    if y_prob.min() < 0:
-        raise ValueError("y_prob contains values less than 0.")
-
-    return label_binarize(y_true, labels)[:, 0]
-
-
 def brier_score_loss(y_true, y_prob, sample_weight=None, pos_label=None):
     """Compute the Brier score.
     The smaller the Brier score, the better, hence the naming with "loss".
@@ -2353,8 +2334,10 @@ def brier_score_loss(y_true, y_prob, sample_weight=None, pos_label=None):
         Sample weights.
 
     pos_label : int or str, default=None
-        Label of the positive class. If None, the maximum label is used as
-        positive class
+        Label of the positive class.
+        When ``pos_label=None``, if y_true is in {-1, 1} or {0, 1},
+        ``pos_label`` is set to 1,
+        otherwise ``pos_label`` is set to the greater label.
 
     Returns
     -------
@@ -2389,8 +2372,28 @@ def brier_score_loss(y_true, y_prob, sample_weight=None, pos_label=None):
     assert_all_finite(y_prob)
     check_consistent_length(y_true, y_prob, sample_weight)
 
+    labels = np.unique(y_true)
+    if len(labels) > 2:
+        raise ValueError("Only binary classification is supported. "
+                         "Provided labels %s." % labels)
+    if y_prob.max() > 1:
+        raise ValueError("y_prob contains values greater than 1.")
+    if y_prob.min() < 0:
+        raise ValueError("y_prob contains values less than 0.")
+
     if pos_label is None:
-        pos_label = y_true.max()
+        if (np.array_equal(labels, [0, 1]) or
+                np.array_equal(labels, [-1, 1]) or
+                np.array_equal(labels, [0]) or
+                np.array_equal(labels, [-1]) or
+                np.array_equal(labels, [1])):
+            # this is the same as the definition of pos_label=None
+            # in roc_curve and precision_recall_curve
+            pos_label = 1
+        else:
+            # this is different from the definition of pos_label=None
+            # in roc_curve and precision_recall_curve
+            # the purpose is to keep backward compatibility
+            pos_label = y_true.max()
     y_true = np.array(y_true == pos_label, int)
-    y_true = _check_binary_probabilistic_predictions(y_true, y_prob)
     return np.average((y_true - y_prob) ** 2, weights=sample_weight)
diff --git a/sklearn/metrics/ranking.py b/sklearn/metrics/ranking.py
index 52404e856135f..7d1d727a36a20 100644
--- a/sklearn/metrics/ranking.py
+++ b/sklearn/metrics/ranking.py
@@ -469,13 +469,16 @@ def precision_recall_curve(y_true, probas_pred, pos_label=None,
     Parameters
     ----------
     y_true : array, shape = [n_samples]
-        True targets of binary classification in range {-1, 1} or {0, 1}.
+        True binary labels. If labels are not either {-1, 1} or {0, 1}, then
+        pos_label should be explicitly given.
 
     probas_pred : array, shape = [n_samples]
         Estimated probabilities or decision function.
 
     pos_label : int or str, default=None
-        The label of the positive class
+        The label of the positive class.
+        When ``pos_label=None``, if y_true is in {-1, 1} or {0, 1},
+        ``pos_label`` is set to 1, otherwise an error will be raised.
 
     sample_weight : array-like of shape = [n_samples], optional
         Sample weights.
@@ -552,7 +555,9 @@ def roc_curve(y_true, y_score, pos_label=None, sample_weight=None,
         (as returned by "decision_function" on some classifiers).
 
     pos_label : int or str, default=None
-        Label considered as positive and others are considered negative.
+        The label of the positive class.
+        When ``pos_label=None``, if y_true is in {-1, 1} or {0, 1},
+        ``pos_label`` is set to 1, otherwise an error will be raised.
 
     sample_weight : array-like of shape = [n_samples], optional
         Sample weights.
diff --git a/sklearn/metrics/tests/test_classification.py b/sklearn/metrics/tests/test_classification.py
index df192ef19b146..d7686fb8589b6 100644
--- a/sklearn/metrics/tests/test_classification.py
+++ b/sklearn/metrics/tests/test_classification.py
@@ -1997,9 +1997,23 @@ def test_brier_score_loss():
     assert_raises(ValueError, brier_score_loss, y_true, y_pred[1:])
     assert_raises(ValueError, brier_score_loss, y_true, y_pred + 1.)
     assert_raises(ValueError, brier_score_loss, y_true, y_pred - 1.)
-    # calculate even if only single class in y_true (#6980)
-    assert_almost_equal(brier_score_loss([0], [0.5]), 0.25)
-    assert_almost_equal(brier_score_loss([1], [0.5]), 0.25)
+
+    # ensure to raise an error for multiclass y_true
+    y_true = np.array([0, 1, 2, 0])
+    y_pred = np.array([0.8, 0.6, 0.4, 0.2])
+    error_message = ("Only binary classification is supported. Provided "
+                     "labels {}".format(np.array([0, 1, 2])))
+    assert_raise_message(ValueError, error_message, brier_score_loss,
+                         y_true, y_pred)
+
+    # calculate correctly when there's only one class in y_true
+    assert_almost_equal(brier_score_loss([-1], [0.4]), 0.16)
+    assert_almost_equal(brier_score_loss([0], [0.4]), 0.16)
+    assert_almost_equal(brier_score_loss([1], [0.4]), 0.36)
+    assert_almost_equal(
+        brier_score_loss(['foo'], [0.4], pos_label='bar'), 0.16)
+    assert_almost_equal(
+        brier_score_loss(['foo'], [0.4], pos_label='foo'), 0.36)
 
 
 def test_balanced_accuracy_score_unseen():


Code review comments:
diff_hunk: @@ -2353,8 +2334,10 @@ def brier_score_loss(y_true, y_prob, sample_weight=None, pos_label=None):
         Sample weights.
 
     pos_label : int or str, default=None
-        Label of the positive class. If None, the maximum label is used as
-        positive class
+        Label of the positive class.
+        When ``pos_label=None``, if y_true is in {-1, 1} or {0, 1},
path: sklearn/metrics/classification.py
text: @user1:
Defaults to the greater label unless y_true is all 0 or all -1 in which case pos_label defaults to 1.

How about that?