Repository: Project-MONAI/MONAI

Issue:
Generalize Metric to any loss function
**Is your feature request related to a problem? Please describe.**
If I introduce a new loss function, I have to change the metric as well. For example: `monai.losses.Dice` and `monai.metrics.DiceMetric`. Most of the losses have not a metric associated.

**Describe the solution you'd like**
I wonder if the metrics can be made general to any loss function. Where the loss function would be an argument of the metric. Other metric arguments would be the reduction method and similar things.

**Describe alternatives you've considered**
The alternative is to develop my own Metric every time I change the loss function. But not ideal imo.


Generalize Metric to any loss function
**Is your feature request related to a problem? Please describe.**
If I introduce a new loss function, I have to change the metric as well. For example: `monai.losses.Dice` and `monai.metrics.DiceMetric`. Most of the losses have not a metric associated.

**Describe the solution you'd like**
I wonder if the metrics can be made general to any loss function. Where the loss function would be an argument of the metric. Other metric arguments would be the reduction method and similar things.

**Describe alternatives you've considered**
The alternative is to develop my own Metric every time I change the loss function. But not ideal imo.




Discussion:
Hi, I am facing a similar issue. For my case I would like to have two or more metrics (eg-Dice, HD95) along with dice as loss. 
However,if both the metrics are computed the loss after couple of epochs results in NaN.

Would love to find out how and if the metrics could be generalized with any loss as @phcerdan mentioned.
great suggestion, the main engineering value-added for monai metric is the feature of computing results from mini-batches and aggregate (potentially over multiple processes) these results at the epoch level (e.g. https://github.com/Project-MONAI/tutorials/blob/master/modules/compute_metric.py). I think you can use the `CumulativeAverage` to achieve averaging for the monai loss functions:
https://github.com/Project-MONAI/MONAI/blob/c2018cf620a7ddf2130a272bf8a1b68ced8f2e42/monai/metrics/cumulative_average.py#L30-L36
the cumulative results could be used for tensorboard plotting/early stopping. So the cumulative APIs ([avg](https://github.com/Project-MONAI/MONAI/blob/c2018cf620a7ddf2130a272bf8a1b68ced8f2e42/monai/metrics/cumulative_average.py#L20), [generic](https://github.com/Project-MONAI/MONAI/blob/c2018cf620a7ddf2130a272bf8a1b68ced8f2e42/monai/metrics/metric.py#L112-L115)) is the generic metric wrapper that you are looking for. Or, is there any specific metric use case that difficult to achieve with the current API?
@wyli Thanks for getting back!
More than early stopping, I would like to use multiple metrics for reporting. For example, use Dice loss along with DSC metric for validation and saving the best model but at the same time make use of one or more additional metric for reporting/ comparison and/or ranking  as specified in  [here](https://github.com/emrekavur/CHAOS-evaluation/blob/master/CHAOS_Metrics_and_Evaluation_Method.pdf)

Example, after each epoch,
Dice = 0.8635, HD95 = 12.2, ASSD = 8.3 (for reporting)

But on computing the extra metrics the loss goes to nan after a couple of epoch.

Normal- 
_epoch 2 average loss: 0.6035
saved new best metric model
current epoch: 2 current mean dice: 0.3898 BG: 0.6600 fem: 0.1438 pel: 0.3656
best mean dice: 0.3898 at epoch: 2
current epoch: 2 current mean hd95: 94.7535 BG: 54.7478 fem: 124.4526 pel: 105.0602
current epoch: 2 current mean assd: 18.3567 BG: 10.2701 fem: 25.0300 pel: 19.7702
time consuming of epoch 2 is: 1576.0210_


Intial error-
_
epoch 11 average loss: 0.5795

/home/umgpy/umgpy/Monai_8_test_env/lib/python3.8/site-packages/monai/metrics/hausdorff_distance.py:160: UserWarning: the prediction of class 0 is all 0, this may result in nan/inf distance.
  warnings.warn(f"the prediction of class {c} is all 0, this may result in nan/inf distance.")
/home/umgpy/umgpy/Monai_8_test_env/lib/python3.8/site-packages/numpy/lib/function_base.py:4486: RuntimeWarning: invalid value encountered in subtract
  diff_b_a = subtract(b, a)
/home/umgpy/umgpy/Monai_8_test_env/lib/python3.8/site-packages/monai/metrics/hausdorff_distance.py:160: UserWarning: the prediction of class 1 is all 0, this may result in nan/inf distance.
  warnings.warn(f"the prediction of class {c} is all 0, this may result in nan/inf distance.")
/home/umgpy/umgpy/Monai_8_test_env/lib/python3.8/site-packages/monai/metrics/hausdorff_distance.py:160: UserWarning: the prediction of class 2 is all 0, this may result in nan/inf distance.
  warnings.warn(f"the prediction of class {c} is all 0, this may result in nan/inf distance.")
/home/umgpy/umgpy/Monai_8_test_env/lib/python3.8/site-packages/monai/metrics/surface_distance.py:155: UserWarning: the prediction of class 0 is all 0, this may result in nan/inf distance.
  warnings.warn(f"the prediction of class {c} is all 0, this may result in nan/inf distance.")
/home/umgpy/umgpy/Monai_8_test_env/lib/python3.8/site-packages/monai/metrics/surface_distance.py:155: UserWarning: the prediction of class 1 is all 0, this may result in nan/inf distance.
  warnings.warn(f"the prediction of class {c} is all 0, this may result in nan/inf distance.")
/umgpy/umgpy/Monai_8_test_env/lib/python3.8/site-packages/monai/metrics/surface_distance.py:155: UserWarning: the prediction of class 2 is all 0, this may result in nan/inf distance.
  warnings.warn(f"the prediction of class {c} is all 0, this may result in nan/inf distance.")

current epoch: 11 current mean dice: 0.1033 BG: 0.1736 fem: 0.0642 pel: 0.0722
best mean dice: 0.4128 at epoch: 6
current epoch: 11 current mean hd95: 89.1498 BG: 39.9831 fem: 141.0331 pel: 86.4331
best mean dice: -1.0000 at epoch: -1
current epoch: 11 current mean assd: inf BG: inf fem: inf pel: inf
best mean dice: -1.0000 at epoch: -1
time consuming of epoch 11 is: 639.5590_


Finally,
_5/8, train_loss: nan, step time: 1.5168
6/8, train_loss: 0.5702, step time: 1.5253
7/8, train_loss: 0.5762, step time: 1.5282
8/8, train_loss: nan, step time: 1.5164
epoch 14 average loss: nan
current epoch: 14 current mean dice: 0.1034 BG: 0.1757 fem: 0.0648 pel: 0.0697
best mean dice: 0.4128 at epoch: 6
current epoch: 14 current mean hd95: 89.3620 BG: 40.0747 fem: 141.1014 pel: 86.9099
current epoch: 14 current mean assd: inf BG: inf fem: inf pel: inf
time consuming of epoch 14 is: 639.2755_

I am using `DiceMetric`, `HausdorffDistanceMetric` and `SurfaceDistanceMetric`. Using them individually is not an issue though.

With regard to using `CumulativeAverage`, since  I do not need all of them averaged it is not much use. Let me know if I am missing something. Would prefer to have additional metrics calculated in a single run than run it multiple times for each of them indivdually.
Thanks 


> But on computing the extra metrics the loss goes to nan after a couple of epoch.

Perhaps you can detach the prediction to prevent backprop before computing alternative loss values?
> So the cumulative APIs ([avg](https://github.com/Project-MONAI/MONAI/blob/c2018cf620a7ddf2130a272bf8a1b68ced8f2e42/monai/metrics/cumulative_average.py#L20), [generic](https://github.com/Project-MONAI/MONAI/blob/c2018cf620a7ddf2130a272bf8a1b68ced8f2e42/monai/metrics/metric.py#L112-L115)) is the generic metric wrapper that you are looking for.

It is indeed, thanks @wyli.
I still fail to fully understand the purpose of `DiceMetric`. Is there any difference between `DiceMetric` and doing a `DiceLoss` + `CumulativeAverage`? Thanks!
Hi, I am facing a similar issue. For my case I would like to have two or more metrics (eg-Dice, HD95) along with dice as loss. 
However,if both the metrics are computed the loss after couple of epochs results in NaN.

Would love to find out how and if the metrics could be generalized with any loss as @phcerdan mentioned.
great suggestion, the main engineering value-added for monai metric is the feature of computing results from mini-batches and aggregate (potentially over multiple processes) these results at the epoch level (e.g. https://github.com/Project-MONAI/tutorials/blob/master/modules/compute_metric.py). I think you can use the `CumulativeAverage` to achieve averaging for the monai loss functions:
https://github.com/Project-MONAI/MONAI/blob/c2018cf620a7ddf2130a272bf8a1b68ced8f2e42/monai/metrics/cumulative_average.py#L30-L36
the cumulative results could be used for tensorboard plotting/early stopping. So the cumulative APIs ([avg](https://github.com/Project-MONAI/MONAI/blob/c2018cf620a7ddf2130a272bf8a1b68ced8f2e42/monai/metrics/cumulative_average.py#L20), [generic](https://github.com/Project-MONAI/MONAI/blob/c2018cf620a7ddf2130a272bf8a1b68ced8f2e42/monai/metrics/metric.py#L112-L115)) is the generic metric wrapper that you are looking for. Or, is there any specific metric use case that difficult to achieve with the current API?
@wyli Thanks for getting back!
More than early stopping, I would like to use multiple metrics for reporting. For example, use Dice loss along with DSC metric for validation and saving the best model but at the same time make use of one or more additional metric for reporting/ comparison and/or ranking  as specified in  [here](https://github.com/emrekavur/CHAOS-evaluation/blob/master/CHAOS_Metrics_and_Evaluation_Method.pdf)

Example, after each epoch,
Dice = 0.8635, HD95 = 12.2, ASSD = 8.3 (for reporting)

But on computing the extra metrics the loss goes to nan after a couple of epoch.

Normal- 
_epoch 2 average loss: 0.6035
saved new best metric model
current epoch: 2 current mean dice: 0.3898 BG: 0.6600 fem: 0.1438 pel: 0.3656
best mean dice: 0.3898 at epoch: 2
current epoch: 2 current mean hd95: 94.7535 BG: 54.7478 fem: 124.4526 pel: 105.0602
current epoch: 2 current mean assd: 18.3567 BG: 10.2701 fem: 25.0300 pel: 19.7702
time consuming of epoch 2 is: 1576.0210_


Intial error-
_
epoch 11 average loss: 0.5795

/home/umgpy/umgpy/Monai_8_test_env/lib/python3.8/site-packages/monai/metrics/hausdorff_distance.py:160: UserWarning: the prediction of class 0 is all 0, this may result in nan/inf distance.
  warnings.warn(f"the prediction of class {c} is all 0, this may result in nan/inf distance.")
/home/umgpy/umgpy/Monai_8_test_env/lib/python3.8/site-packages/numpy/lib/function_base.py:4486: RuntimeWarning: invalid value encountered in subtract
  diff_b_a = subtract(b, a)
/home/umgpy/umgpy/Monai_8_test_env/lib/python3.8/site-packages/monai/metrics/hausdorff_distance.py:160: UserWarning: the prediction of class 1 is all 0, this may result in nan/inf distance.
  warnings.warn(f"the prediction of class {c} is all 0, this may result in nan/inf distance.")
/home/umgpy/umgpy/Monai_8_test_env/lib/python3.8/site-packages/monai/metrics/hausdorff_distance.py:160: UserWarning: the prediction of class 2 is all 0, this may result in nan/inf distance.
  warnings.warn(f"the prediction of class {c} is all 0, this may result in nan/inf distance.")
/home/umgpy/umgpy/Monai_8_test_env/lib/python3.8/site-packages/monai/metrics/surface_distance.py:155: UserWarning: the prediction of class 0 is all 0, this may result in nan/inf distance.
  warnings.warn(f"the prediction of class {c} is all 0, this may result in nan/inf distance.")
/home/umgpy/umgpy/Monai_8_test_env/lib/python3.8/site-packages/monai/metrics/surface_distance.py:155: UserWarning: the prediction of class 1 is all 0, this may result in nan/inf distance.
  warnings.warn(f"the prediction of class {c} is all 0, this may result in nan/inf distance.")
/umgpy/umgpy/Monai_8_test_env/lib/python3.8/site-packages/monai/metrics/surface_distance.py:155: UserWarning: the prediction of class 2 is all 0, this may result in nan/inf distance.
  warnings.warn(f"the prediction of class {c} is all 0, this may result in nan/inf distance.")

current epoch: 11 current mean dice: 0.1033 BG: 0.1736 fem: 0.0642 pel: 0.0722
best mean dice: 0.4128 at epoch: 6
current epoch: 11 current mean hd95: 89.1498 BG: 39.9831 fem: 141.0331 pel: 86.4331
best mean dice: -1.0000 at epoch: -1
current epoch: 11 current mean assd: inf BG: inf fem: inf pel: inf
best mean dice: -1.0000 at epoch: -1
time consuming of epoch 11 is: 639.5590_


Finally,
_5/8, train_loss: nan, step time: 1.5168
6/8, train_loss: 0.5702, step time: 1.5253
7/8, train_loss: 0.5762, step time: 1.5282
8/8, train_loss: nan, step time: 1.5164
epoch 14 average loss: nan
current epoch: 14 current mean dice: 0.1034 BG: 0.1757 fem: 0.0648 pel: 0.0697
best mean dice: 0.4128 at epoch: 6
current epoch: 14 current mean hd95: 89.3620 BG: 40.0747 fem: 141.1014 pel: 86.9099
current epoch: 14 current mean assd: inf BG: inf fem: inf pel: inf
time consuming of epoch 14 is: 639.2755_

I am using `DiceMetric`, `HausdorffDistanceMetric` and `SurfaceDistanceMetric`. Using them individually is not an issue though.

With regard to using `CumulativeAverage`, since  I do not need all of them averaged it is not much use. Let me know if I am missing something. Would prefer to have additional metrics calculated in a single run than run it multiple times for each of them indivdually.
Thanks 


> But on computing the extra metrics the loss goes to nan after a couple of epoch.

Perhaps you can detach the prediction to prevent backprop before computing alternative loss values?
> So the cumulative APIs ([avg](https://github.com/Project-MONAI/MONAI/blob/c2018cf620a7ddf2130a272bf8a1b68ced8f2e42/monai/metrics/cumulative_average.py#L20), [generic](https://github.com/Project-MONAI/MONAI/blob/c2018cf620a7ddf2130a272bf8a1b68ced8f2e42/monai/metrics/metric.py#L112-L115)) is the generic metric wrapper that you are looking for.

It is indeed, thanks @wyli.
I still fail to fully understand the purpose of `DiceMetric`. Is there any difference between `DiceMetric` and doing a `DiceLoss` + `CumulativeAverage`? Thanks!

Resolving patch:
diff --git a/monai/metrics/__init__.py b/monai/metrics/__init__.py
--- a/monai/metrics/__init__.py
+++ b/monai/metrics/__init__.py
@@ -16,6 +16,7 @@
 from .froc import compute_fp_tp_probs, compute_froc_curve_data, compute_froc_score
 from .generalized_dice import GeneralizedDiceScore, compute_generalized_dice
 from .hausdorff_distance import HausdorffDistanceMetric, compute_hausdorff_distance, compute_percent_hausdorff_distance
+from .loss_metric import LossMetric
 from .meandice import DiceMetric, compute_dice, compute_meandice
 from .meaniou import MeanIoU, compute_iou, compute_meaniou
 from .metric import Cumulative, CumulativeIterationMetric, IterationMetric, Metric
diff --git a/monai/metrics/loss_metric.py b/monai/metrics/loss_metric.py
new file mode 100644
--- /dev/null
+++ b/monai/metrics/loss_metric.py
@@ -0,0 +1,106 @@
+# Copyright (c) MONAI Consortium
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#     http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from typing import Union
+
+import torch
+from torch.nn.modules.loss import _Loss
+
+from monai.metrics.utils import do_metric_reduction
+from monai.utils import MetricReduction
+
+from .metric import CumulativeIterationMetric
+
+
+class LossMetric(CumulativeIterationMetric):
+    """
+    A wrapper to make ``loss_fn`` available as a cumulative metric. That is, the loss values computed from
+    mini-batches can be combined in the ``reduction`` mode across multiple iterations, as a quantitative measurement
+    of a model.
+
+    Example:
+
+    .. code-block:: python
+
+        import torch
+        from monai.losses import DiceLoss
+        from monai.metrics import LossMetric
+
+        dice_loss = DiceLoss(include_background=True)
+        loss_metric = LossMetric(loss_fn=dice_loss)
+
+        # first iteration
+        y_pred = torch.tensor([[[[1.0, 0.0], [0.0, 1.0]]]])  # shape [batch=1, channel=1, 2, 2]
+        y = torch.tensor([[[[1.0, 0.0], [1.0, 1.0]]]])  # shape [batch=1, channel=1, 2, 2]
+        loss_metric(y_pred, y)
+
+        # second iteration
+        y_pred = torch.tensor([[[[1.0, 0.0], [0.0, 0.0]]]])  # shape [batch=1, channel=1, 2, 2]
+        y = torch.tensor([[[[1.0, 0.0], [1.0, 1.0]]]])  # shape [batch=1, channel=1, 2, 2]
+        loss_metric(y_pred, y)
+
+        # aggregate
+        print(loss_metric.aggregate(reduction="none"))  # tensor([[0.2000], [0.5000]]) (shape [batch=2, channel=1])
+
+        # reset
+        loss_metric.reset()
+        print(loss_metric.aggregate())
+
+
+    Args:
+        loss_fn: a callable function that takes ``y_pred`` and optionally ``y`` as input (in the "batch-first" format),
+            returns a "batch-first" tensor of loss values.
+        reduction: define mode of reduction to the metrics, will only apply reduction on `not-nan` values,
+            available reduction modes: {``"none"``, ``"mean"``, ``"sum"``, ``"mean_batch"``, ``"sum_batch"``,
+            ``"mean_channel"``, ``"sum_channel"``}, default to ``"mean"``. if "none", will not do reduction.
+        get_not_nans: whether to return the `not_nans` count, if True, aggregate() returns (metric, not_nans).
+            Here `not_nans` count the number of not nans for the metric, thus its shape equals to the shape of the metric.
+
+    """
+
+    def __init__(
+        self, loss_fn: _Loss, reduction: Union[MetricReduction, str] = MetricReduction.MEAN, get_not_nans: bool = False
+    ) -> None:
+        super().__init__()
+        self.loss_fn = loss_fn
+        self.reduction = reduction
+        self.get_not_nans = get_not_nans
+
+    def aggregate(self, reduction: Union[MetricReduction, str, None] = None):
+        """
+        Returns the aggregated loss value across multiple iterations.
+
+        Args:
+            reduction: define mode of reduction to the metrics, will only apply reduction on `not-nan` values,
+                available reduction modes: {``"none"``, ``"mean"``, ``"sum"``, ``"mean_batch"``, ``"sum_batch"``,
+                ``"mean_channel"``, ``"sum_channel"``}, default to `self.reduction`. if "none", will not do reduction.
+        """
+        data = self.get_buffer()
+        if data is None:
+            return (torch.tensor(0.0), torch.tensor(0.0)) if self.get_not_nans else torch.tensor(0.0)
+        f, not_nans = do_metric_reduction(data, reduction or self.reduction)
+        return (f, not_nans) if self.get_not_nans else f
+
+    def _compute_tensor(self, y_pred: torch.Tensor, y: torch.Tensor = None):  # type: ignore
+        """
+        Input `y_pred` is compared with ground truth `y`.
+        Both `y_pred` and `y` are expected to be a batch-first Tensor (BC[HWD]).
+
+        Returns:
+             a tensor with shape (BC[HWD]), or a list of tensors, each tensor with shape (C[HWD]).
+        """
+        iter_loss = self.loss_fn(y_pred) if y is None else self.loss_fn(y_pred, y)
+        if isinstance(iter_loss, torch.Tensor):
+            while iter_loss.dim() < 2:
+                iter_loss = iter_loss[None]
+        # to be compatible with `Cumulative`, iter_loss should at least have a batch dim.
+        # to be compatible with `do_metric_reduction`, iter_loss should at least have a batch and a channel dim.
+        return iter_loss
diff --git a/monai/metrics/metric.py b/monai/metrics/metric.py
--- a/monai/metrics/metric.py
+++ b/monai/metrics/metric.py
@@ -202,7 +202,7 @@ def extend(self, *data) -> None:
         for b, d in zip(self._buffers, data):
             # converting to pytorch tensors so that we can use the distributed API
             d_t, *_ = convert_data_type(d, output_type=torch.Tensor, wrap_sequence=True)
-            try:
+            try:  # d_t must be a mini-batch of values
                 b.extend([x[0] for x in torch.split(d_t, 1, dim=0)])
             except (AttributeError, IndexError, RuntimeError) as e:
                 raise TypeError(
@@ -286,6 +286,7 @@ class CumulativeIterationMetric(Cumulative, IterationMetric):
 
     Typically, it computes some intermediate results for each iteration, adds them to the buffers,
     then the buffer contents could be gathered and aggregated for the final result when epoch completed.
+    Currently,``Cumulative.aggregate()`` and ``IterationMetric._compute_tensor()`` are expected to be implemented.
 
     For example, `MeanDice` inherits this class and the usage is as follows:
 
@@ -324,7 +325,8 @@ def __call__(self, y_pred: TensorOrList, y: Optional[TensorOrList] = None):
                 or a `batch-first` Tensor.
 
         Returns:
-            The computed metric values at the iteration level.
+            The computed metric values at the iteration level. The output shape should be
+            a `batch-first` tensor (BC[HWD]) or a list of `batch-first` tensors.
         """
         ret = super().__call__(y_pred=y_pred, y=y)
         if isinstance(ret, (tuple, list)):


Test patch:
diff --git a/tests/test_loss_metric.py b/tests/test_loss_metric.py
new file mode 100644
--- /dev/null
+++ b/tests/test_loss_metric.py
@@ -0,0 +1,54 @@
+# Copyright (c) MONAI Consortium
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#     http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import unittest
+
+import numpy as np
+import torch
+from parameterized import parameterized
+
+from monai.losses import DiceLoss
+from monai.metrics import LossMetric
+
+_device = "cuda:0" if torch.cuda.is_available() else "cpu"
+TEST_CASE_1 = [  # y (1, 1, 2, 2), y_pred (1, 1, 2, 2), expected out (1, 1)
+    {
+        "loss_class": DiceLoss,
+        "loss_kwargs": {"include_background": True},
+        "reduction": "mean",
+        "get_not_nans": False,
+        "y_pred": torch.tensor([[[[1.0, 0.0], [0.0, 1.0]]]], device=_device),
+        "y": torch.tensor([[[[1.0, 0.0], [1.0, 1.0]]]], device=_device),
+        "include_background": True,
+    },
+    [0.2],
+]
+
+
+class TestComputeLossMetric(unittest.TestCase):
+    @parameterized.expand([TEST_CASE_1])
+    def test_value_class(self, input_data, expected_value):
+        loss_fn = input_data["loss_class"](**input_data["loss_kwargs"])
+        loss_metric = LossMetric(
+            loss_fn=loss_fn, reduction=input_data["reduction"], get_not_nans=input_data["get_not_nans"]
+        )
+
+        loss_metric(y_pred=input_data.get("y_pred"), y=input_data.get("y"))
+        loss_metric(y_pred=input_data.get("y_pred"), y=input_data.get("y"))
+        result = loss_metric.aggregate()
+        np.testing.assert_allclose(result.cpu().numpy(), expected_value, atol=1e-4)
+        loss_metric.reset()
+        result = loss_metric.aggregate()
+        np.testing.assert_allclose(result.cpu().numpy(), 0.0, atol=1e-4)
+
+
+if __name__ == "__main__":
+    unittest.main()