Repository: scikit-learn/scikit-learn
Pull request: ENH Add warning when only one label found in `confusion_matrix`
#### Reference Issues/PRs
fixes #19756

#### What does this implement/fix? Explain your changes.
Add warning when only one label found in `confusion_matrix` and advise user to pass all labels via `labels` parameter.



Problem statement:
Confusion Matrix is of wrong shape


#### Describe the bug

Having 100 percent true positive rate gives a 1x1 matrix. This can cause bugs if you are using the confusion matrix in a function to calculate evaluation parameters such as sensitivity, auc, specifity, accuracy.

```
from sklearn.metrics import confusion_matrix, roc_auc_score

ground_truth, predictions = np.ones((2,1))
cf_matrix = confusion_matrix(ground_truth, predictions)
assert cf_matrix.shape == (2,2), ("The confusion matrix is of shape"+str(cf_matrix.shape))
```

#### Expected Results
No error is thrown.

#### Actual Results
```
AssertionError                            Traceback (most recent call last)
<ipython-input-23-9c79b721fccb> in <module>
      3 ground_truth, predictions = np.ones((2,1))
      4 cf_matrix = confusion_matrix(ground_truth, predictions)
----> 5 assert cf_matrix.shape == (2,2), ("The confusion matrix is of shape"+str(cf_matrix.shape))

AssertionError: The confusion matrix is of shape(1, 1). 
```
#### Versions
```
System:
    python: 3.7.10 | packaged by conda-forge | (default, Feb 19 2021, 15:59:12)  [Clang 11.0.1 ]
executable: /Users/julian/opt/anaconda3/envs/cadd-course/bin/python
   machine: Darwin-20.3.0-x86_64-i386-64bit

Python dependencies:
          pip: 21.0.1
   setuptools: 49.6.0.post20210108
      sklearn: 0.24.1
        numpy: 1.20.1
        scipy: 1.6.1
       Cython: None
       pandas: 1.2.3
   matplotlib: 3.3.4
       joblib: 1.0.1
threadpoolctl: 2.1.0

Built with OpenMP: True
```



<!-- Thanks for contributing! -->

Hints from issue discussion:
If `y_true` and `y_pred` does not contain all the known labels, then you can use `labels` to pass them in:

```py
from sklearn.metrics import confusion_matrix, roc_auc_score

ground_truth, predictions = np.ones((2,1))
cf_matrix = confusion_matrix(ground_truth, predictions, labels=[0, 1])
cf_matrix
# array([[0, 0],
#        [0, 1]])
```
Maybe we should raise an error or a warning when only one label is found in the input?
Okay that works. However, I suggest for usability: 
1.) Checking if we have that case with 
```
    if all(y_true == y_pred):
        warnings.warn("The model has 100% true positive rate")
        cm=np.zeros((2, 2))
        cm[0][0]+=sum(y_pred == 0)
        cm[1][1]+=sum(y_pred == 1)
    else:
        cm = coo_matrix((sample_weight, (y_true, y_pred)),
                        shape=(n_labels, n_labels), dtype=dtype,
                        ).toarray()

        with np.errstate(all='ignore'):
            if normalize == 'true':
                cm = cm / cm.sum(axis=1, keepdims=True)
            elif normalize == 'pred':
                cm = cm / cm.sum(axis=0, keepdims=True)
            elif normalize == 'all':
                cm = cm / cm.sum()
            cm = np.nan_to_num(cm)

    return cm
```
in metrics/_classification.py

2.) Also adding to the metrics/tests/test_classification.py: 
```
ground_truth, predictions=np.ones((2, 1))
cf_matrix=confusion_matrix(ground_truth, predictions)
assert cf_matrix.shape == (2, 2), ("The confusion matrix is of shape " + str(cf_matrix.shape))
```
We can't catch every such case without making labels mandatory. A holistic
solution is to have a tool which generates appropriate metric functions or
scorers for a given complete (not split/sampled) dataset (#12385), and to
encourage its use to establish metrics upfront given a task rather than
using metric functions and scorers ad hoc.

It is a case which requires special attention on many levels. Therefore, handling it, is absolutely mandatory. 
> It is a case which requires special attention on many levels. Therefore, handling it, is absolutely mandatory.

Can you bit more explicit by what do you mean by "this case"?
I agree with @jnothman that it is not as easy as it seems since we don't have any knowledge of the classification problem to be solved (binary or multiclass). Without #12385, issuing a warning as proposed by @NicolasHug might the most sensible thing to do, mentioning that providing `labels` would silence the warning and provide the confusion matrix with the expected shape.

Commit message: black

Patch to review:
diff --git a/sklearn/metrics/_classification.py b/sklearn/metrics/_classification.py
index d1dc3232c53ab..73260a2be1d52 100644
--- a/sklearn/metrics/_classification.py
+++ b/sklearn/metrics/_classification.py
@@ -382,6 +382,15 @@ def confusion_matrix(
             cm = cm / cm.sum()
         cm = np.nan_to_num(cm)
 
+    if cm.shape == (1, 1):
+        warnings.warn(
+            (
+                "Only one label was found in 'y_true' and 'y_pred', use the 'labels' "
+                "parameter to pass all known labels."
+            ),
+            UserWarning,
+        )
+
     return cm
 
 


Code review comments:
diff_hunk: @@ -382,6 +382,15 @@ def confusion_matrix(
             cm = cm / cm.sum()
         cm = np.nan_to_num(cm)
 
+    if cm.shape == (1, 1):
+        warnings.warn(
+            (
+                "Only one label was found in 'y_true' and 'y_pred', use the 'labels' "
path: sklearn/metrics/_classification.py
text: @user1:
```suggestion
                "A single label is found in 'y_true' and 'y_pred', use the 'labels' "
```

@author:
This is nicer!