Repository: pytorch/ignite

Issue:
Documentation for ConfusionMatrix is thin on details, and error messages not informative
## 🐛 Bug description

Confusion matrix requires that `y_pred` and `y` are to have different dimensions `(batch,num_classes)` and `(batch,)`, respectively. Getting this wrong results in the following error message, which makes no sense (batch size is 5 here, and both `y_pred` and `y` have shape `(5,1)`:

`y_pred must have shape (batch_size, num_categories, ...) and y must have shape of (batch_size, ...), but given torch.Size([5, 1]) vs torch.Size([5, 1])`

I believe that the `torch.Size([5, 1]) vs torch.Size([5, 1])` message should probably read `torch.Size([5, 1]) vs torch.Size([5,])` to illustrate the problem.

Separately, it would be good to mention in the `ConfusionMatrix` docs that one-hot encoded vectors are required, and that this can be implemented using `from ignite.utils.to_onehot` (I stumbled upon it after realizing I had to one-hot encode, since most people use `BCEWithLogitsLoss()` anyway, and hence their model predictions are not one-hot encoded).

## Environment

 - PyTorch Version (e.g., 1.4): 1.4.0
 - Ignite Version (e.g., 0.3.0): 0.4.2
 - OS (e.g., Linux): MacOS
 - How you installed Ignite (`conda`, `pip`, source): `pip`
 - Python version: 3.8
 - Any other relevant information:



Discussion:
Thanks for the report @nsinenian ! 
Context : https://discuss.pytorch.org/t/ignite-confusionmatrix-metric-error/99079
@vfdev-5 Hi, Could you take me through this code? Any link? Thanks.
@Afzal-Ind please take a look here: https://github.com/pytorch/ignite/blob/master/ignite/metrics/confusion_matrix.py
> Separately, it would be good to mention in the ConfusionMatrix docs that one-hot encoded vectors are required, and that this can be implemented using from ignite.utils.to_onehot (I stumbled upon it after realizing I had to one-hot encode, since most people use BCEWithLogitsLoss() anyway, and hence their model predictions are not one-hot encoded).

@nsinenian actually, this is not exactly true. Predictions are required to have shape `(batch_size, num_categories, ...)` (logits or after softmax), one-hot encoded. And i'd say, this something we directly have as network output for classification/segmentation tasks. As for targets, they should not be one-hot encoded: `(batch_size, ...)`
https://github.com/pytorch/ignite/blob/dca273fc73cdeefe0bf6858de64c69f23cbffe8b/ignite/metrics/confusion_matrix.py#L17-L19
@vfdev-5 The comment was with respect to predictions, not targets. For the predictions, I have (relevant excerpt):
````
__NUM_CLASSES__ = 2

def binary_one_hot_output_transform(output):
    """ Ouptuts a one-hot encoded vector with 0 or 1 values """
    [y_pred,y] = output
    [y_pred,y] = [torch.sigmoid(y_pred).round().long(),y]    
    return (to_onehot(y_pred,__NUM_CLASSES__), y.long())

valuation_metrics = {
        'confusion_matrix' : ConfusionMatrix(num_classes=__NUM_CLASSES__,output_transform=binary_one_hot_output_transform),
}

evaluator = create_supervised_evaluator(model,metrics=valuation_metrics,output_transform=lambda x,y,y_pred: (y_pred,y))
trainer = create_supervised_trainer(model,optimizer,criterion)

````
My model output (forward function) is:
````
    def forward(self,features):
      
        y = self.fc1(features)
        y = self.fcd(y)
        y = self.fc2(F.relu(y))

        return y
````

For binary classification, I use the `BCEWithLogitsLoss()` loss function which does not use one-hot encoded vectors (and hence the model output being what it is above). Perhaps my use case is atypical.
@nsinenian thanks for the details. Yes, that's true that you have binary classification use-case and yes, it definitely makes sense to mention that how to deal with that in the docs !
As far as I can see, the issue is only with binary classification, right?

I'd suggest changing the [docstring](https://pytorch.org/ignite/_modules/ignite/metrics/confusion_matrix.html) from

    - `y_pred` must contain logits and has the following shape (batch_size, num_categories, ...).

to

    - `y_pred` must contain logits and has the following shape (batch_size, num_categories, ...). If you are doing binary classification, you may first need to one-hot encode the network output, e.g. with `ignite.utils.to_onehot(torch.sigmoid(y_pred).round().long(), 2)`.

@vfdev-5 Ok for that?

@jrieke I'd say the following have to be done here:
- add a note to `num_classes` arg that it should be > 1
- add this check into the code and raise ValueError otherwise
- add a basic test for that
- analyze the code and see if we still can have a message like 
```
y_pred must have shape (batch_size, num_categories, ...) and y must have shape of (batch_size, ...), but given torch.Size([5, 1]) vs torch.Size([5, 1])
```
if yes, make the message more user-friendly.
- add a more explicit note for the binary case. Maybe, a bit more polished version of this code snippet: https://github.com/pytorch/ignite/issues/1380#issuecomment-727253899 

What do you think ?

Resolving patch:
diff --git a/ignite/metrics/confusion_matrix.py b/ignite/metrics/confusion_matrix.py
--- a/ignite/metrics/confusion_matrix.py
+++ b/ignite/metrics/confusion_matrix.py
@@ -14,13 +14,14 @@ class ConfusionMatrix(Metric):
     """Calculates confusion matrix for multi-class data.
 
     - ``update`` must receive output of the form ``(y_pred, y)`` or ``{'y_pred': y_pred, 'y': y}``.
-    - `y_pred` must contain logits and has the following shape (batch_size, num_categories, ...)
+    - `y_pred` must contain logits and has the following shape (batch_size, num_classes, ...).
+      If you are doing binary classification, see Note for an example on how to get this.
     - `y` should have the following shape (batch_size, ...) and contains ground-truth class indices
       with or without the background class. During the computation, argmax of `y_pred` is taken to determine
       predicted classes.
 
     Args:
-        num_classes (int): number of classes. See notes for more details.
+        num_classes (int): Number of classes, should be > 1. See notes for more details.
         average (str, optional): confusion matrix values averaging schema: None, "samples", "recall", "precision".
             Default is None. If `average="samples"` then confusion matrix values are normalized by the number of seen
             samples. If `average="recall"` then confusion matrix values are normalized such that diagonal values
@@ -39,6 +40,27 @@ class ConfusionMatrix(Metric):
         contribute to the confusion matrix and others are neglected. For example, if `num_classes=20` and target index
         equal 255 is encountered, then it is filtered out.
 
+        If you are doing binary classification with a single output unit, you may have to transform your network output,
+        so that you have one value for each class. E.g. you can transform your network output into a one-hot vector
+        with:
+
+        .. code-block:: python
+
+            def binary_one_hot_output_transform(output):
+                y_pred, y = output
+                y_pred = torch.sigmoid(y_pred).round().long()
+                y_pred = ignite.utils.to_onehot(y_pred, 2)
+                y = y.long()
+                return y_pred, y
+
+            metrics = {
+                "confusion_matrix": ConfusionMatrix(2, output_transform=binary_one_hot_output_transform),
+            }
+
+            evaluator = create_supervised_evaluator(
+                model, metrics=metrics, output_transform=lambda x, y, y_pred: (y_pred, y)
+            )
+
     """
 
     def __init__(
@@ -51,6 +73,9 @@ def __init__(
         if average is not None and average not in ("samples", "recall", "precision"):
             raise ValueError("Argument average can None or one of 'samples', 'recall', 'precision'")
 
+        if num_classes <= 1:
+            raise ValueError("Argument num_classes needs to be > 1")
+
         self.num_classes = num_classes
         self._num_examples = 0
         self.average = average
@@ -65,17 +90,18 @@ def _check_shape(self, output: Sequence[torch.Tensor]) -> None:
         y_pred, y = output[0].detach(), output[1].detach()
 
         if y_pred.ndimension() < 2:
-            raise ValueError(f"y_pred must have shape (batch_size, num_categories, ...), but given {y_pred.shape}")
-
-        if y_pred.shape[1] != self.num_classes:
             raise ValueError(
-                f"y_pred does not have correct number of categories: {y_pred.shape[1]} vs {self.num_classes}"
+                f"y_pred must have shape (batch_size, num_classes (currently set to {self.num_classes}), ...), "
+                f"but given {y_pred.shape}"
             )
 
+        if y_pred.shape[1] != self.num_classes:
+            raise ValueError(f"y_pred does not have correct number of classes: {y_pred.shape[1]} vs {self.num_classes}")
+
         if not (y.ndimension() + 1 == y_pred.ndimension()):
             raise ValueError(
-                "y_pred must have shape (batch_size, num_categories, ...) and y must have "
-                "shape of (batch_size, ...), "
+                f"y_pred must have shape (batch_size, num_classes (currently set to {self.num_classes}), ...) "
+                "and y must have shape of (batch_size, ...), "
                 f"but given {y.shape} vs {y_pred.shape}."
             )
 


Test patch:
diff --git a/tests/ignite/metrics/test_confusion_matrix.py b/tests/ignite/metrics/test_confusion_matrix.py
--- a/tests/ignite/metrics/test_confusion_matrix.py
+++ b/tests/ignite/metrics/test_confusion_matrix.py
@@ -19,17 +19,27 @@ def test_no_update():
         cm.compute()
 
 
+def test_num_classes_wrong_input():
+    with pytest.raises(ValueError, match="Argument num_classes needs to be > 1"):
+        ConfusionMatrix(num_classes=1)
+
+
 def test_multiclass_wrong_inputs():
     cm = ConfusionMatrix(10)
 
-    with pytest.raises(ValueError, match=r"y_pred must have shape \(batch_size, num_categories, ...\)"):
+    with pytest.raises(
+        ValueError, match=r"y_pred must have shape \(batch_size, num_classes " r"\(currently set to 10\), ...\)"
+    ):
         cm.update((torch.rand(10), torch.randint(0, 2, size=(10,)).long()))
 
-    with pytest.raises(ValueError, match=r"y_pred does not have correct number of categories:"):
+    with pytest.raises(ValueError, match=r"y_pred does not have correct number of classes:"):
         cm.update((torch.rand(10, 5, 4), torch.randint(0, 2, size=(10,)).long()))
 
     with pytest.raises(
-        ValueError, match=r"y_pred must have shape \(batch_size, num_categories, ...\) " r"and y must have "
+        ValueError,
+        match=r"y_pred must have shape \(batch_size, num_classes "
+        r"\(currently set to 10\), ...\) "
+        r"and y must have ",
     ):
         cm.update((torch.rand(4, 10, 12, 12), torch.randint(0, 10, size=(10,)).long()))