Repository: Lightning-AI/torchmetrics

Issue:
Jaccard Index behaviour when num_classes is absent
The [jaccard index function](https://github.com/PyTorchLightning/metrics/blob/8fade87062a7b87c1e6429bbe1c4e0112b3713a5/torchmetrics/functional/classification/jaccard.py#L69) determines the number of classes automatically.
This can result in strange results if the user forgets to specify `num_classes`. For example, in IoU determination in semantic segmentation, not all images have all classes. So determining them automatically will almost always lead to wrong number.
Additionally, the [warning](https://github.com/PyTorchLightning/metrics/blob/8fade87062a7b87c1e6429bbe1c4e0112b3713a5/torchmetrics/utilities/data.py#L156) that shows up makes it seem like something is wrong, when the user specifies the number of classes explicitly (but this is not the case at all):

```python
/run/determined/pythonuserbase/lib/python3.7/site-packages/torchmetrics/utilities/prints.py:36: RuntimeWarning: You have set 11 number of classes which is different from predicted (1000) and target (10) number of classes
```

Would it be crazy if `num_classes` requirement is default and the automatic determination of classes is a special option that must be enabled?




Discussion:
Hi! thanks for your contribution!, great first issue!
@SkafteNicki @twsl thoughts? :rabbit: 
Its fine by me to update this to be a required argument.
@drozzy mind sending a PR? :rabbit: 

Resolving patch:
diff --git a/torchmetrics/functional/classification/jaccard.py b/torchmetrics/functional/classification/jaccard.py
--- a/torchmetrics/functional/classification/jaccard.py
+++ b/torchmetrics/functional/classification/jaccard.py
@@ -69,10 +69,10 @@ def _jaccard_from_confmat(
 def jaccard_index(
     preds: Tensor,
     target: Tensor,
+    num_classes: int,
     ignore_index: Optional[int] = None,
     absent_score: float = 0.0,
     threshold: float = 0.5,
-    num_classes: Optional[int] = None,
     reduction: str = "elementwise_mean",
 ) -> Tensor:
     r"""
@@ -95,6 +95,7 @@ def jaccard_index(
     Args:
         preds: tensor containing predictions from model (probabilities, or labels) with shape ``[N, d1, d2, ...]``
         target: tensor containing ground truth labels with shape ``[N, d1, d2, ...]``
+        num_classes: Specify the number of classes
         ignore_index: optional int specifying a target class to ignore. If given,
             this class index does not contribute to the returned score, regardless
             of reduction method. Has no effect if given an int that is not in the
@@ -107,8 +108,7 @@ def jaccard_index(
             assigned the `absent_score`.
         threshold:
             Threshold value for binary or multi-label probabilities.
-        num_classes:
-            Optionally specify the number of classes
+
         reduction: a method to reduce metric score over labels.
 
             - ``'elementwise_mean'``: takes the mean (default)
@@ -124,7 +124,7 @@ def jaccard_index(
         >>> target = torch.randint(0, 2, (10, 25, 25))
         >>> pred = torch.tensor(target)
         >>> pred[2:5, 7:13, 9:15] = 1 - pred[2:5, 7:13, 9:15]
-        >>> jaccard_index(pred, target)
+        >>> jaccard_index(pred, target, num_classes=2)
         tensor(0.9660)
     """
 


Test patch:
diff --git a/tests/classification/test_jaccard.py b/tests/classification/test_jaccard.py
--- a/tests/classification/test_jaccard.py
+++ b/tests/classification/test_jaccard.py
@@ -155,6 +155,7 @@ def test_jaccard(half_ones, reduction, ignore_index, expected):
     jaccard_val = jaccard_index(
         preds=preds,
         target=target,
+        num_classes=3,
         ignore_index=ignore_index,
         reduction=reduction,
     )
@@ -233,3 +234,14 @@ def test_jaccard_ignore_index(pred, target, ignore_index, num_classes, reduction
         reduction=reduction,
     )
     assert torch.allclose(jaccard_val, tensor(expected).to(jaccard_val))
+
+
+def test_warning_on_difference_in_number_of_classes():
+    """Test that warning is thrown if the detected number of classes are different from the the specified number of
+    classes."""
+    preds = torch.randint(3, (10,))
+    target = torch.randint(3, (10,))
+    with pytest.warns(
+        RuntimeWarning,
+    ):
+        jaccard_index(preds, target, num_classes=4)