Repository: Lightning-AI/torchmetrics

Issue:
[metrics] AUROC Metric can't handle 0 observations of a class with multiclass classifier
I'm attempting to calculate AUROC for a multiclass problem where some classes are very rare, occasionally never seen, and I'm getting the following error: `raise ValueError("No positive samples in targets, true positive value should be meaningless")`

In the case of 0 observations, I feel the `average='weighted'` should work, since the contribution to the final AUROC should be 0 regardless. One can think of other scenarios where there are a very high number of classes, some of which will happen to not be seen in some dataset.

_Originally posted by @BeyondTheProof in https://github.com/PyTorchLightning/pytorch-lightning/issues/2210#issuecomment-872440776_


Discussion:
I've found a hack for this by subclassing AUROC:
```py
class FixedAUC(torchmetrics.AUROC):
    def update(self, preds: torch.Tensor, target: torch.Tensor):
        num_classes = preds.shape[1]
        zero_obs_mask = torch.tensor([(target == c).sum() > 0 for c in range(num_classes)])
        preds = preds[:, zero_obs_mask]
        target = target[:, zero_obs_mask]
        self.num_classes = zero_obs_mask.sum().int()
        super().update(torch.softmax(preds, axis=-1).data, target.argmax(axis=-1).int().data)
```

I will be making a cleaner fix in torch and asking for a PR
Hi! thanks for your contribution!, great first issue!
Hi @BeyondTheProof, I transferred the issue to the torchmetrics repo.
Can I kindly ask if you are calling the `forward` method or the `update` method of the metric when trying to update the metric states?
Hi @SkafteNicki, thank you for the response and transferring the issue! I am just calling the `update` method.
Could you provide an example of how you are using the metric?
In my subclassed lightning module, I have metrics that are calculated at the end of each epoch:

```
self.auroc = torchmetrics.AUROC(num_classes=15, average="weighted", compute_on_step=False)
self.accuracy = pl.metrics.Accuracy(num_classes=15, average="weighted", compute_on_step=False)
self.metrics = [self.auroc, self.accuracy]
```

When I calculate all my losses, I also calculate some other metrics (if there are any bugs, it is only because I modified it to simplify as much as possible):
```
for metric, name in zip(self.metrics, ["AUROC", "ACC"]):
  self.log(
      f"{stage}_{name}",
      metric,
      on_step=False,
      on_epoch=True,
      prog_bar=False,
      logger=True,
  )
```

I hope this helps. If there's anything else you need, please let me know!
> I've found a hack for this by subclassing AUROC:
> 
> ```python
> class FixedAUC(torchmetrics.AUROC):
>     def update(self, preds: torch.Tensor, target: torch.Tensor):
>         num_classes = preds.shape[1]
>         zero_obs_mask = torch.tensor([(target == c).sum() > 0 for c in range(num_classes)])
>         preds = preds[:, zero_obs_mask]
>         target = target[:, zero_obs_mask]
>         self.num_classes = zero_obs_mask.sum().int()
>         super().update(torch.softmax(preds, axis=-1).data, target.argmax(axis=-1).int().data)
> ```
> 
> I will be making a cleaner fix in torch and asking for a PR

@SkafteNicki In this subclass, there is an error that arises when some batches have only two classes, others have more. I implemented a much cleaner subclass by just subclassing Metrics:

```
from torchmetrics.functional.classification.auroc import _auroc_compute


class WeightedAUROC(Metric):
    """
    This is used for when the target is not strictly one of K classes, but a probability
    distribution over all K classes
    """

    def __init__(
        self,
        num_classes: Optional[int] = None,
        pos_label: Optional[int] = None,
        compute_on_step: bool = False,
        dist_sync_on_step: bool = False,
    ) -> None:
        super().__init__(
            compute_on_step=compute_on_step,
            dist_sync_on_step=dist_sync_on_step,
            process_group=process_group,
        )
        self.num_classes = num_classes
        if self.num_classes > 2:
            self.mode = "multiclass"
        else:
            self.mode = "binary"
        self.pos_label = pos_label
        self.average = "weighted"

        self.add_state("preds", default=[], dist_reduce_fx="cat")
        self.add_state("target", default=[], dist_reduce_fx="cat")

    def update(self, preds: torch.Tensor, target: torch.Tensor):
        assert target.ndim == 2, f"WeightedAUC expects a 2D Tensor, got {target.ndim}"
        self.preds.append(torch.softmax(preds, axis=-1).data)
        self.target.append(target.data)

    def compute(self):
        preds = dim_zero_cat(self.preds)
        target = dim_zero_cat(self.target)
        zero_obs_mask = torch.tensor([(target == c).sum() > 0 for c in range(self.num_classes)])
        preds = preds[:, zero_obs_mask]
        target = target[:, zero_obs_mask].int()
        num_classes = zero_obs_mask.sum().int()
        return _auroc_compute(preds, target, self.mode, num_classes=num_classes, average=self.average)
```
@BeyondTheProof I thinks that it will be better to add the line `process_group: Optional[Any] = None,`.
```from torchmetrics.functional.classification.auroc import _auroc_compute


class WeightedAUROC(Metric):
    """
    This is used for when the target is not strictly one of K classes, but a probability
    distribution over all K classes
    """

    def __init__(
        self,
        num_classes: Optional[int] = None,
        pos_label: Optional[int] = None,
        compute_on_step: bool = False,
        dist_sync_on_step: bool = False,
        process_group: Optional[Any] = None, # It seems that this line is missing.
    ) -> None:
        super().__init__(
            compute_on_step=compute_on_step,
            dist_sync_on_step=dist_sync_on_step,
            process_group=process_group,
        )
        self.num_classes = num_classes
        if self.num_classes > 2:
            self.mode = "multiclass"
        else:
            self.mode = "binary"
        self.pos_label = pos_label
        self.average = "weighted"

        self.add_state("preds", default=[], dist_reduce_fx="cat")
        self.add_state("target", default=[], dist_reduce_fx="cat")

    def update(self, preds: torch.Tensor, target: torch.Tensor):
        assert target.ndim == 2, f"WeightedAUC expects a 2D Tensor, got {target.ndim}"
        self.preds.append(torch.softmax(preds, axis=-1).data)
        self.target.append(target.data)

    def compute(self):
        preds = dim_zero_cat(self.preds)
        target = dim_zero_cat(self.target)
        zero_obs_mask = torch.tensor([(target == c).sum() > 0 for c in range(self.num_classes)])
        preds = preds[:, zero_obs_mask]
        target = target[:, zero_obs_mask].int()
        num_classes = zero_obs_mask.sum().int()
        return _auroc_compute(preds, target, self.mode, num_classes=num_classes, average=self.average)
```
Hi @BeyondTheProof, thanks for getting back to me. It indeed seems to be a problem, even though I would consider this a very corner case. Would you be up for sending a fix to our current AUROC implementation? 
IMO the only thing missing from your implementation is that the user should be warned if a class was removed.



Hi @SkafteNicki, good point about the user warning. Also, in the implementation above, the target only allows 2D tensors, i.e., a `(obs, classes)` binary matrix. I will implement a solution for this as well.
> @BeyondTheProof I thinks that it will be better to add the line `process_group: Optional[Any] = None,`.

Thanks for the catch @KimDaeUng, you're totally right!

Resolving patch:
diff --git a/torchmetrics/functional/classification/auroc.py b/torchmetrics/functional/classification/auroc.py
--- a/torchmetrics/functional/classification/auroc.py
+++ b/torchmetrics/functional/classification/auroc.py
@@ -11,6 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
+import warnings
 from typing import Optional, Sequence, Tuple
 
 import torch
@@ -87,8 +88,23 @@ def _auroc_compute(
         else:
             raise ValueError('Detected input to be `multilabel` but you did not provide `num_classes` argument')
     else:
-        if mode != DataType.BINARY and num_classes is None:
-            raise ValueError('Detected input to `multiclass` but you did not provide `num_classes` argument')
+        if mode != DataType.BINARY:
+            if num_classes is None:
+                raise ValueError("Detected input to `multiclass` but you did not provide `num_classes` argument")
+            if average == AverageMethod.WEIGHTED and len(torch.unique(target)) < num_classes:
+                # If one or more classes has 0 observations, we should exclude them, as its weight will be 0
+                target_bool_mat = torch.zeros((len(target), num_classes), dtype=bool)
+                target_bool_mat[torch.arange(len(target)), target.long()] = 1
+                class_observed = target_bool_mat.sum(axis=0) > 0
+                for c in range(num_classes):
+                    if not class_observed[c]:
+                        warnings.warn(f'Class {c} had 0 observations, omitted from AUROC calculation', UserWarning)
+                preds = preds[:, class_observed]
+                target = target_bool_mat[:, class_observed]
+                target = torch.where(target)[1]
+                num_classes = class_observed.sum()
+                if num_classes == 1:
+                    raise ValueError('Found 1 non-empty class in `multiclass` AUROC calculation')
         fpr, tpr, _ = roc(preds, target, num_classes, pos_label, sample_weights)
 
     # calculate standard roc auc score


Test patch:
diff --git a/tests/classification/test_auroc.py b/tests/classification/test_auroc.py
--- a/tests/classification/test_auroc.py
+++ b/tests/classification/test_auroc.py
@@ -195,3 +195,31 @@ def test_error_multiclass_no_num_classes():
         ValueError, match="Detected input to `multiclass` but you did not provide `num_classes` argument"
     ):
         _ = auroc(torch.randn(20, 3).softmax(dim=-1), torch.randint(3, (20, )))
+
+
+def test_weighted_with_empty_classes():
+    """ Tests that weighted multiclass AUROC calculation yields the same results if a new
+        but empty class exists. Tests that the proper warnings and errors are raised
+    """
+    preds = torch.tensor([
+        [0.90, 0.05, 0.05],
+        [0.05, 0.90, 0.05],
+        [0.05, 0.05, 0.90],
+        [0.85, 0.05, 0.10],
+        [0.10, 0.10, 0.80],
+    ])
+    target = torch.tensor([0, 1, 1, 2, 2])
+    num_classes = 3
+    _auroc = auroc(preds, target, average="weighted", num_classes=num_classes)
+
+    # Add in a class with zero observations at second to last index
+    preds = torch.cat((preds[:, :num_classes - 1], torch.rand_like(preds[:, 0:1]), preds[:, num_classes - 1:]), axis=1)
+    # Last class (2) gets moved to 3
+    target[target == num_classes - 1] = num_classes
+    with pytest.warns(UserWarning, match='Class 2 had 0 observations, omitted from AUROC calculation'):
+        _auroc_empty_class = auroc(preds, target, average="weighted", num_classes=num_classes + 1)
+    assert _auroc == _auroc_empty_class
+
+    target = torch.zeros_like(target)
+    with pytest.raises(ValueError, match='Found 1 non-empty class in `multiclass` AUROC calculation'):
+        _ = auroc(preds, target, average="weighted", num_classes=num_classes + 1)