Repository: Project-MONAI/MONAI

Issue:
Multi-class 2D segmentation: merged classes in prediction
Hello, everyone!

I recently discovered MONAI and I am trying to get it to work on some of our projects. I really like the modularity of the library, but I fear that I am still confused with some of the steps. In particular, I am now facing a strange issue with the results of the network predictions.

I adapted the example from https://github.com/Project-MONAI/tutorials/blob/master/3d_segmentation/brats_segmentation_3d.ipynb for a 2D case of segmentation of cells. As ground truth for the training, I am using a (3-channel, one-hot) mask with one channel for the background pixels, one for the cell bodies and one for the cell boundaries (see the details below). The prediction seems to work more or less okay, with the exception that channel 0 contains the background pixels (as expected), channel 1 contains the cell-body pixels (as expected), and channel 2 contains cell-body + edge pixels (see second panel from the right in the figure below). If I subtract channel 1 from channel 2, I get a clean cell boundary signal (right-most panel), but I am confused as the reason why is the interior filled.

## Data
* Images are gray-value, one channel. Size of the tensor (without the batch dimension) is (1, 700, 1100). Intensities are normalized and `torch.float32`.
* Masks are one-hot (3 channel, binary), with channel 0 being the background class, channel 1 the cell bodies, and channel 2 the cell boundaries. Size of the tensor (without the batch dimension) is (3, 700, 1100). Type is `float32`.

A ROI is applied during training and validation/prediction (see below).

## Hyperparameters

Training and predictions are run on a `(256, 256)` ROI.

### Training

The architecture of the network is the same as in the example, with the exception of setting `dimensions=2` and `in_channels=1`. As in the example, `out_channels=3`.

```python
loss_function = DiceLoss(include_background=False, to_onehot_y=False, sigmoid=True, squared_pred=True)
optimizer = Adam(model.parameters(), 1e-3, weight_decay=1e-4, amsgrad=True)
````
In contrast to the 3D example, I added `include_background=False` to the loss function and increased the learning rate and decay by one order of magnitude.

### Validation

I set `include_background=False` for the metric calculation. The rest is as in the example.

```python
dice_metric = DiceMetric(include_background=False, reduction="mean")
post_trans = Compose([Activations(sigmoid=True), AsDiscrete(threshold_values=True)])
```

### Prediction

Same changes as in the validation.

```python
post_trans = Compose([Activations(sigmoid=True), AsDiscrete(threshold_values=True)])
```

Is there any obvious reason why prediction of channel 2 seems to accumulate the predictions of expected channels 1 and 2? 

This is a small inset of a prediction that shows the issue:

![prediction](https://user-images.githubusercontent.com/1226043/100360035-b03b4000-2ff8-11eb-92c1-f7e2bc700b66.png)

**Legend**

* `0`: prediction of background pixels;
* `1`: prediction of cell-body pixels;
* `2`: prediction of cell-boundary pixels;
* `2 - 1`: predicted channel 1 subtracted from predicted channel 2 (pretty close to ground truth, not shown).

Thanks a lot for any suggestions!
Aaron
Multi-class 2D segmentation: merged classes in prediction
Hello, everyone!

I recently discovered MONAI and I am trying to get it to work on some of our projects. I really like the modularity of the library, but I fear that I am still confused with some of the steps. In particular, I am now facing a strange issue with the results of the network predictions.

I adapted the example from https://github.com/Project-MONAI/tutorials/blob/master/3d_segmentation/brats_segmentation_3d.ipynb for a 2D case of segmentation of cells. As ground truth for the training, I am using a (3-channel, one-hot) mask with one channel for the background pixels, one for the cell bodies and one for the cell boundaries (see the details below). The prediction seems to work more or less okay, with the exception that channel 0 contains the background pixels (as expected), channel 1 contains the cell-body pixels (as expected), and channel 2 contains cell-body + edge pixels (see second panel from the right in the figure below). If I subtract channel 1 from channel 2, I get a clean cell boundary signal (right-most panel), but I am confused as the reason why is the interior filled.

## Data
* Images are gray-value, one channel. Size of the tensor (without the batch dimension) is (1, 700, 1100). Intensities are normalized and `torch.float32`.
* Masks are one-hot (3 channel, binary), with channel 0 being the background class, channel 1 the cell bodies, and channel 2 the cell boundaries. Size of the tensor (without the batch dimension) is (3, 700, 1100). Type is `float32`.

A ROI is applied during training and validation/prediction (see below).

## Hyperparameters

Training and predictions are run on a `(256, 256)` ROI.

### Training

The architecture of the network is the same as in the example, with the exception of setting `dimensions=2` and `in_channels=1`. As in the example, `out_channels=3`.

```python
loss_function = DiceLoss(include_background=False, to_onehot_y=False, sigmoid=True, squared_pred=True)
optimizer = Adam(model.parameters(), 1e-3, weight_decay=1e-4, amsgrad=True)
````
In contrast to the 3D example, I added `include_background=False` to the loss function and increased the learning rate and decay by one order of magnitude.

### Validation

I set `include_background=False` for the metric calculation. The rest is as in the example.

```python
dice_metric = DiceMetric(include_background=False, reduction="mean")
post_trans = Compose([Activations(sigmoid=True), AsDiscrete(threshold_values=True)])
```

### Prediction

Same changes as in the validation.

```python
post_trans = Compose([Activations(sigmoid=True), AsDiscrete(threshold_values=True)])
```

Is there any obvious reason why prediction of channel 2 seems to accumulate the predictions of expected channels 1 and 2? 

This is a small inset of a prediction that shows the issue:

![prediction](https://user-images.githubusercontent.com/1226043/100360035-b03b4000-2ff8-11eb-92c1-f7e2bc700b66.png)

**Legend**

* `0`: prediction of background pixels;
* `1`: prediction of cell-body pixels;
* `2`: prediction of cell-boundary pixels;
* `2 - 1`: predicted channel 1 subtracted from predicted channel 2 (pretty close to ground truth, not shown).

Thanks a lot for any suggestions!
Aaron


Discussion:
Hi @wyli ,

Could you please help with this question as you are a research expert on model training?

Thanks in advance.

thanks for the detailed report!

it's getting some okay results so I'd assume there's no obvious bug in the training pipeline. according to the setup the class channel 2 has only a few positive examples, compared with class channel 1, perhaps it's difficult to learn a good classifier for that channel. I'd try a few things:
- increase the threshold value for channel 2 in inference `AsDiscrete(threshold_values=True)` (default is at 0.5)
- training with `softmax=True` instead of `sigmoid=True`, that'll take into account the fact that the three channels are mutually exclusive
- try the `GeneralizedDiceLoss` which may better handle the class imbalance
- perhaps the boundary vs interior discrimination is inherently very difficult, could consider some preprocessing transforms to enhance the boundary intensity before training

Thanks for your feedback! I'll try and post my results here.
A few results and observations:

* Changing `sigmoid=True` to `softmax=True` in the loss function indeed did the trick! The cell body and cell boundary classes are not neatly separated.
* To handle the large class imbalance, I tried switching to `GeneralizedDiceLoss` as suggested, but I get the following error (I tried both `monai-weekly` and the latest commit from the repository):

```python
Traceback (most recent call last):
  File ".../Segmenter.py", line 298, in <module>
    loss = loss_function(outputs, labels)
  File ".../envs/deep_tools/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File ".../envs/deep_tools/lib/python3.8/site-packages/monai/losses/dice.py", line 345, in forward
    f: torch.Tensor = 1.0 - (2.0 * (intersection * w).sum(1) + self.smooth_nr) / (
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)
```
Is there some preparation step that needs to be performed before the output of the forward pass is ready for the `GeneralizedDiceLoss` function?

Thanks!
that looks like an issue in the loss function, could you try `batch=False` in GeneralizedDiceLoss
Hi @wyli ,

Could you please help with this question as you are a research expert on model training?

Thanks in advance.

thanks for the detailed report!

it's getting some okay results so I'd assume there's no obvious bug in the training pipeline. according to the setup the class channel 2 has only a few positive examples, compared with class channel 1, perhaps it's difficult to learn a good classifier for that channel. I'd try a few things:
- increase the threshold value for channel 2 in inference `AsDiscrete(threshold_values=True)` (default is at 0.5)
- training with `softmax=True` instead of `sigmoid=True`, that'll take into account the fact that the three channels are mutually exclusive
- try the `GeneralizedDiceLoss` which may better handle the class imbalance
- perhaps the boundary vs interior discrimination is inherently very difficult, could consider some preprocessing transforms to enhance the boundary intensity before training

Thanks for your feedback! I'll try and post my results here.
A few results and observations:

* Changing `sigmoid=True` to `softmax=True` in the loss function indeed did the trick! The cell body and cell boundary classes are not neatly separated.
* To handle the large class imbalance, I tried switching to `GeneralizedDiceLoss` as suggested, but I get the following error (I tried both `monai-weekly` and the latest commit from the repository):

```python
Traceback (most recent call last):
  File ".../Segmenter.py", line 298, in <module>
    loss = loss_function(outputs, labels)
  File ".../envs/deep_tools/lib/python3.8/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File ".../envs/deep_tools/lib/python3.8/site-packages/monai/losses/dice.py", line 345, in forward
    f: torch.Tensor = 1.0 - (2.0 * (intersection * w).sum(1) + self.smooth_nr) / (
IndexError: Dimension out of range (expected to be in range of [-1, 0], but got 1)
```
Is there some preparation step that needs to be performed before the output of the forward pass is ready for the `GeneralizedDiceLoss` function?

Thanks!
that looks like an issue in the loss function, could you try `batch=False` in GeneralizedDiceLoss

Resolving patch:
diff --git a/monai/losses/dice.py b/monai/losses/dice.py
--- a/monai/losses/dice.py
+++ b/monai/losses/dice.py
@@ -342,8 +342,8 @@ def forward(self, input: torch.Tensor, target: torch.Tensor) -> torch.Tensor:
             b[infs] = 0.0
             b[infs] = torch.max(b)
 
-        f: torch.Tensor = 1.0 - (2.0 * (intersection * w).sum(1) + self.smooth_nr) / (
-            (denominator * w).sum(1) + self.smooth_dr
+        f: torch.Tensor = 1.0 - (2.0 * (intersection * w).sum(0 if self.batch else 1) + self.smooth_nr) / (
+            (denominator * w).sum(0 if self.batch else 1) + self.smooth_dr
         )
 
         if self.reduction == LossReduction.MEAN.value:


Test patch:
diff --git a/tests/test_generalized_wasserstein_dice_loss.py b/tests/test_generalized_wasserstein_dice_loss.py
--- a/tests/test_generalized_wasserstein_dice_loss.py
+++ b/tests/test_generalized_wasserstein_dice_loss.py
@@ -135,7 +135,7 @@ def test_convergence(self):
         to segment one image.
         We verify that the loss is decreasing in almost all SGD steps.
         """
-        learning_rate = 0.01
+        learning_rate = 0.001
         max_iter = 50
 
         # define a simple 3d example
@@ -174,8 +174,8 @@ def forward(self, x):
             # initialize the loss
             loss = GeneralizedWassersteinDiceLoss(dist_matrix=np.array([[0.0, 1.0], [1.0, 0.0]]), weighting_mode=w_mode)
 
-            # initialize an SGD
-            optimizer = optim.SGD(net.parameters(), lr=learning_rate, momentum=0.9)
+            # initialize an optimizer
+            optimizer = optim.Adam(net.parameters(), lr=learning_rate)
 
             # initial difference between pred and target
             pred_start = torch.argmax(net(image), dim=1)
diff --git a/tests/test_seg_loss_integration.py b/tests/test_seg_loss_integration.py
--- a/tests/test_seg_loss_integration.py
+++ b/tests/test_seg_loss_integration.py
@@ -31,6 +31,7 @@
     [GeneralizedDiceLoss, {"to_onehot_y": True, "sigmoid": True}, {}],
     [GeneralizedDiceLoss, {"to_onehot_y": True, "sigmoid": True, "w_type": "simple"}, {}],
     [GeneralizedDiceLoss, {"to_onehot_y": True, "sigmoid": True, "w_type": "uniform"}, {}],
+    [GeneralizedDiceLoss, {"to_onehot_y": True, "sigmoid": True, "w_type": "uniform", "batch": True}, {}],
     [TverskyLoss, {"to_onehot_y": True, "softmax": True, "alpha": 0.8, "beta": 0.2}, {}],
     [TverskyLoss, {"to_onehot_y": True, "softmax": True, "alpha": 0.8, "beta": 0.2, "batch": True}, {}],
     [TverskyLoss, {"to_onehot_y": True, "softmax": True, "alpha": 1.0, "beta": 0.0}, {}],