Repository: Project-MONAI/MONAI

Issue:
Training instability with Dice Loss/Tversky Loss
I am training a 2D UNet to segment fetal MR images using MONAI and I have been observing some instability in the training when using MONAI Dice loss formulation. After some iteration, the loss jumps up and the network stops learning, as the gradients drop to zero. Here is an example (orange is loss on training set computed over 2D slices, blue is loss on validation computed over 3D volume).
![image](https://user-images.githubusercontent.com/28106739/88265546-9b0a7980-ccc5-11ea-9158-a716dafd32c7.png)

After investigating several aspects (using the same deterministic seed), I've narrowed down the issue to the presence of the ```smooth``` term in both the numerator and denominator of the Dice Loss:
```f = 1.0 - (2.0 * intersection + smooth) / (denominator + smooth)```

When using the formulation:
```f = 1.0 - (2.0 * intersection) / (denominator + smooth)```
without the smooth term in the numerator, the training was stable and no longer showed unexpected behaviour:
![image](https://user-images.githubusercontent.com/28106739/88266174-a27e5280-ccc6-11ea-8b5b-ec14c008a92f.png)
[Note: this experiment was trained for much longer to make sure the jump would not appear later in the training]

The same pattern was observed also for the Tversky Loss, so it could be worth investigating the stability of the losses to identify the best default option.

**Software version**
MONAI version: 0.1.0+84.ga683c4e.dirty
Python version: 3.7.4 (default, Jul  9 2019, 03:52:42)  [GCC 5.4.0 20160609]
Numpy version: 1.18.2
Pytorch version: 1.4.0
Ignite version: 0.3.0

**Training information**
Using MONAI PersistentCache 
2D UNet (as default in MONAI)
Adam optimiser, LR = 1e-3, no LR decay
Batch size: 10

**Other tests**
The following aspects were investigated but did **not** solve the instability issue:
- Gradient clipping
- Different optimisers (SGD, SGD + Momentum)
- Transforming the binary segmentations to a two-channel approach ([background segmentation, foreground segmentation])
- Choosing ```smooth = 1.0``` as default here (https://github.com/MIC-DKFZ/nnUNet/blob/master/nnunet/training/loss_functions/dice_loss.py). However, this made the behaviour even more severe and the jump would happen sooner in the training.

The following losses were also investigated
- Binary Cross Entropy --> stable
- Dice Loss + Binary Cross Entropy --> unstable
- Dice Loss (no smooth at numerator) + Binary Cross Entropy --> stable
- Tversky Loss --> Unstable
- Tversky Loss (no smooth at numerator) --> stable




Discussion:
thanks for this nice report, will look into this (cc @FabianIsensee @ericspod @holgerroth)
Interesting observation. I must admit that I have never investigated this in depth in nnU-Net. I should also note that while the loss function defaults to `smooth=1`, nnU-Net actually uses `smooth=1e-5` (see here https://github.com/MIC-DKFZ/nnUNet/blob/9524dc33425627d1e4b21336a5202b1bcc8157e5/nnunet/training/network_training/nnUNetTrainer.py#L112).

Have you tried training the 2D configuration of nnU-Net with this dataset? It would be quite interesting in knowing whether the instability happens there as well
No, I actually haven't tried and trained nnUnet with my data, but I can look into it in the next days and feedback whether I observe the same behaviour :) (@LucasFidon may find this conversation useful as well)
When you do, please beware that there is a memory leak on Ubuntu 18 and above with Turing cards. Unfortunately there is nothing I can do about it because this is related to mixed precision training with apex/amp (and also occurs with pytorch autocast). I already opened an issue about this and I hope someone will look into this. If you observe this on your system as well, please let me know. You can get rid of it by training in fp32 with the `-fp32` option (you will need more GPU memory). 

Let me know if you need any assistance getting 2D data to run. This is not straightforward right now. You basically need to create dummy 3D niftis that have shape 1 in the first axis.

Best,
Fabian

I'm trying to replicate this issue with some synthetic data, looks like using `sigmoid=True` instead of `softmax=True` is much more stable. @martaranzini are you using the softmax=True option, and could you please try sigmoid=True if possible?

i guess softmax is causing some over/underflow I'll look further into this
Hi @wyli 
I was actually using `sigmoid=True` already in the experiments I reported in the issue. When using a single-channel output, I put `sigmoid=True` and `softmax=False`. Conversely, in the two-channel test I did, I used the softmax instead of the sigmoid. In both cases I would observe the reported behaviour.

We are also working on training the nnU-net on the same data to see if it shows the same behaviour. We are going to report back about it as soon as possible. 
ok, thanks for the information, I haven't tested a single-channel case yet, I'll do that as well. for now two-channel + sigmoid seems to be more stable compared with two-channel + softmax. looks like it's independent of the choice of network afaik
Ok, I see, thanks! I can try and test two-channel + sigmoid instead of softmax on my data as well and see if it gets more stable. I will let you know.
Hi all,

First, apologies for the delay in getting back to you about this issue – we have been running a few experiments to put together a MONAI and nnU-Net comparison and this required quite some time. We hope you will find our results interesting and informative.

@LucasFidon kindly ran a few experiments with nnU-Net and this allowed us to identify some implementation differences. Both MONAI and nnU-Net use the Dice formulation:
![CodeCogsEqn](https://user-images.githubusercontent.com/28106739/93765463-d4e7f580-fc0c-11ea-9f38-052521ac6538.gif)
However, as a default MONAI computes the Dice per element in the batch and then the loss is averaged across the batch (we will refer to this simply as “Dice”). We noticed that in nnU-Net the Dice is instead computed directly as a single value across the whole batch (i.e. not per image). The average is computed only across the channels, not across the batch elements. We refer to this approach as “Batch Dice”. 
We experimented with both formulations in both frameworks, and also tested for Dice loss and Dice + Cross entropy loss at training, as well as for the use of a single- or 2-channel approach. 

The performance of the different trained models on the validation set is reported below (note: they are separated in groups A-E for our own clinical interpretation, but the group separation is not particularly relevant for this issue). The sampling strategy at training is also reported.
![monai_comparison1](https://user-images.githubusercontent.com/28106739/93765510-e92bf280-fc0c-11ea-852f-011322d256f8.png)

A few specifications:

- MONAI – Dice no smooth at numerator used the formulation: 
![CodeCogsEqn-1](https://user-images.githubusercontent.com/28106739/93765562-052f9400-fc0d-11ea-86b3-25b5326ab19b.gif)
- nnU-Net – Batch Dice + Xent, 2-channel, ensemble indicates ensemble performance from 5-fold cross validation at training
- NeuroImage indicates a published two-step approach on our dataset, and it is reported just for reference.

We gather two main observations from these experiments:
1.	**Training instability**: On our dataset, nnU-Net did not present the training instability observed with the default MONAI implementation of Dice. This is also confirmed when known implementation differences were ruled out (nnU-Net – Dice, 2-channel, uniform sampling). Also, nnU-Net generally provides better performance. @FabianIsensee, are there any other implementation differences that could justify our results?
2.	**Dice vs Batch Dice**: In both frameworks, the Batch Dice implementation clearly outperforms the “normal” Dice computation. This could be an interesting feature to be added in MONAI – happy to open another issue/PR about this.

@wyli: I did retrain also the two-channel with sigmoid instead of softmax. With respect to the single-channel, the gradients do not drop to zero (with either sigmoid and softmax), but I still observe some instability in the loss which I cannot fully explain:
![monai_comparison2](https://user-images.githubusercontent.com/28106739/93765815-65bed100-fc0d-11ea-8cda-3d3cc774736c.png)

Looking forward to hearing your comments, and happy to run more experiments to investigate this further!
Hi @martaranzini , thank you so much for the detailed report! It's a very interesting read and I am happy to see that nnU-Net did not disappoint.

Just so that I think about the results in the correct context (@LucasFidon ) : Did you provide separate 2D slices as training examples? Or did you just run the 2D nnU-net configuration on the 3D images? My guess is the latter, but it would be important to be sure.

The batch dice (this is what I call it as well) is implemented on purpose in nnU-Net, but it is not always active. nnU-Net sets that automatically:
- if the patch size does not cover the entire image nnU-Net uses batch dice. Since small patches do not represent the true class distribution anyways we might as well benefit from the increased stability of batch dice. Batch dice is used for all 2D U-Nets or 3d_fullres U-Nets if no 3d_lowres is present
- nnU-Net uses 'sample dice' (that is what I call the regular Dice) for all cases where the patch size covers almost entire training cases, so that would be the 3d_lowres U-Net and the 3d_fullres U-Net if no lowres is present.

Note that there are cases where sample Dice is better than batch dice, so replacing it categorically is not a good idea.

I don't know about the MONAI implementation, but using the sample Dice in 2D segmentation tasks in nnU-Net is a bad idea. That is because nnU-Net *never* optimizes the background class with the Dice (just the CE). What this translates to is a loss function that will not yield any useful gradients on slices where no foreground voxels are present - essentially ignoring these slices in optimization. If you want to run sample Dice with nnU-Net, make sure to set `do_bg=True` in the SoftDiceLoss :-) (this problem does not exist when pairing the Dice with CE because the CE term saves it ;-) )

In my experiments, using uniform sampling gives pretty much the same results as oversampling foreground. Right now we see uniform sampling in nnU-Net only for a scenario which is also unstable in non-uniform sampling. It would be very interesting to see the nnU-Net performance for uniform sampling with the default loss function as well to ensure that oversampling it not skewing the results.

I am not quite sure I understand what you mean by 'ensemble indicates ensemble performance from 5-fold cross validation at training'. Is that simply the Dice score averaged over the five folds? Or do you have a heldout validation set where you applied the ensemble to?

Since your segmentation problem seems to be 3D - what was your motivation to go with a 2D U-Net? I believe you could get better results by running the 3D U-Net. If you have compute resources to spare you could just run the 3d_fullres config. If your dataset is public I can also do that for you :-)

Let me know what you think!

Best,

Fabian


Hi @FabianIsensee, thank you for the quick reply!

Regarding 3D vs 2D, in fetal MRI the 3D MRI are motion-corrupted stacks of 2D slices. In addition the in-plane resolution is typically several times higher than the resolution across planes (please see [1] for more details).
In [1] it has already been shown empirically that 2D CNNs were superior to 3D CNNs for the task of fetal brain extraction. As a result, I have trained nnU-Net only in '2d' mode. For training, the 2D slices were used as input (with an extra first dimension of size 1 as you suggested above).

Regarding the validation dataset used in the figure of @martaranzini, it is a heldout validation set.
The nnU-Net ensemble is the ensemble of the five 2D U-Nets trained for the five folds created by nnUNet.

The dataset is not publicly available... But I will train nnU-Net for the default configuration (batch Dice + CE) except I will deactivate the oversampling :)

Do you think that it is worth trying `do_bg=True` also with batch Dice + CE or does `do_bg=False` always work better in your experience?

Thanks,
Lucas

[1] [Ebner, Michael, et al. "An automated framework for localization, segmentation and super-resolution reconstruction of fetal brain MRI." NeuroImage 206 (2020): 116324.](https://www.sciencedirect.com/science/article/pii/S1053811919309152)
Hi @FabianIsensee ,

Thanks for your comments, very interesting, especially in the analysis of when batch Dice is more suitable than sample Dice. I totally agree that it should not replace it categorically, but it would be useful to add an option in MONAI for the user to choose one over the other.

I will leave to @LucasFidon the answers to the nnU-Net experiments as he actually did all the hard part on that :) 

I can however reply to the details about our segmentation problem. Our task is brain segmentation of fetal MR images for subsequent super-resolution-reconstruction. Our data is 3D but because of heavy (inter-slice) motion artifacts (and heavy anisotropic resolution), we need to approach it as a 2D problem, where we segment each slice independently. Only at inference we perform a 3D assessment, by computing the Dice for the whole 3D image - and this 3D assessment is reported for the validation set in the figure above. We did run some tests with 3D training but it was clearly underperforming the 2D approach. 

In addition, the FOV of our images are quite large and we do have a lot of background-only slices - and we want to be able to correctly predict them as empty. So the comment you raised about the background is spot-on and we will look into greater detail about it. I know for sure we did include the background in MONAI (in the two-channel approaches), but we will check the single-channel and nnU-Net better for this.
Re: the single-channel, I think that removing the smooth term at the numerator in the Dice will have the same effect of ignoring the background slices. On the other hand, we have this instability in the Dice loss and in the Tversky loss in the MONAI framework when keeping the smooth term at the numerator that we cannot explain.

Many thanks!
Marta
Oups, I think we posted about at the same time, thanks for answering the nnU-Net questions, @LucasFidon!
Hi Lucas,

thanks for your response. In that case it would certainly have been easier to simply plug the 3D volumes into nnU-net and let it handle the slicing itself (it's gonna choose the in plane axis automatically and this way you will also get whole 3D volumes as output). I presume you have generated a manual split file to ensure that the data is split properly during cross-validation? This is very important!

I don't think that evaluating `do_bg=True` with batch dice + CE is going to give you a substantially different result. I think it is more important to use it in combination with sample Dice (without CE). CE covers up a lot of the potential shortcomings of the Dice loss, which is why we use it in nnU-Net (nnU-Net is supposed to be as robust as possible). 

nnU-Net handles anisotropic 3D data very well. Motion artifacts can certainly cause a problem, but I would really encourage you to also try the 3D data and see what happens. You might be surprised. Generally speaking you are of course correct, though. On very anisotropic datasets (for example ACDC cine MRI) the benefits of 3D over 2D are less than they could be.

@martaranzini Your response appeared right while I was writing, so let me address your points in this post as well: As long as you are using batch dice OR an additional CE term (essentially in all the cases where nnU-Net did not fail) you do not need to worry about empty slices (as long as they are also presented during training). That becomes only relevant when sample Dice is used without an additional CE term and without `do_bg=True`.

Regarding the smooth term in the numerator: Yes I think of it the same way. If there is no smooth term then the numerator and thus the entire loss is just 0 for empty slices. Note that this is a little bit more complicated though:
- if you are using softmax and you are also optimizing the background class, this should not be a problem because by optimizing the background you are still affecting the output neurons of foreground (coupled via softmax)
- if you are using softmax and not optimizing the background class then this will be a problem
- if you are using sigmoid, this will be a problem (no coupling of outputs, so this is always a problem irrespective of whether you optimize the bg or not)

Best,
Fabian


Resolving patch:
diff --git a/monai/losses/dice.py b/monai/losses/dice.py
--- a/monai/losses/dice.py
+++ b/monai/losses/dice.py
@@ -26,12 +26,12 @@ class DiceLoss(_Loss):
     Compute average Dice loss between two tensors. It can support both multi-classes and multi-labels tasks.
     Input logits `input` (BNHW[D] where N is number of classes) is compared with ground truth `target` (BNHW[D]).
     Axis N of `input` is expected to have logit predictions for each class rather than being image channels,
-    while the same axis of `target` can be 1 or N (one-hot format). The `smooth` parameter is a value added to the
-    intersection and union components of the inter-over-union calculation to smooth results and prevent divide by 0,
-    this value should be small. The `include_background` class attribute can be set to False for an instance of
-    DiceLoss to exclude the first category (channel index 0) which is by convention assumed to be background.
-    If the non-background segmentations are small compared to the total image size they can get overwhelmed by
-    the signal from the background so excluding it in such cases helps convergence.
+    while the same axis of `target` can be 1 or N (one-hot format). The `smooth_nr` and `smooth_dr` parameters are
+    values added to the intersection and union components of the inter-over-union calculation to smooth results
+    respectively, these values should be small. The `include_background` class attribute can be set to False for
+    an instance of DiceLoss to exclude the first category (channel index 0) which is by convention assumed to be
+    background. If the non-background segmentations are small compared to the total image size they can get
+    overwhelmed by the signal from the background so excluding it in such cases helps convergence.
 
     Milletari, F. et. al. (2016) V-Net: Fully Convolutional Neural Networks forVolumetric Medical Image Segmentation, 3DV, 2016.
 
@@ -47,6 +47,9 @@ def __init__(
         squared_pred: bool = False,
         jaccard: bool = False,
         reduction: Union[LossReduction, str] = LossReduction.MEAN,
+        smooth_nr: float = 1e-5,
+        smooth_dr: float = 1e-5,
+        batch: bool = False,
     ) -> None:
         """
         Args:
@@ -66,6 +69,12 @@ def __init__(
                 - ``"mean"``: the sum of the output will be divided by the number of elements in the output.
                 - ``"sum"``: the output will be summed.
 
+            smooth_nr: a small constant added to the numerator to avoid zero.
+            smooth_dr: a small constant added to the denominator to avoid nan.
+            batch: whether to sum the intersection and union areas over the batch dimension before the dividing.
+                Defaults to False, a Dice loss value is computed independently from each item in the batch
+                before any `reduction`.
+
         Raises:
             TypeError: When ``other_act`` is not an ``Optional[Callable]``.
             ValueError: When more than 1 of [``sigmoid=True``, ``softmax=True``, ``other_act is not None``].
@@ -84,13 +93,15 @@ def __init__(
         self.other_act = other_act
         self.squared_pred = squared_pred
         self.jaccard = jaccard
+        self.smooth_nr = float(smooth_nr)
+        self.smooth_dr = float(smooth_dr)
+        self.batch = batch
 
-    def forward(self, input: torch.Tensor, target: torch.Tensor, smooth: float = 1e-5) -> torch.Tensor:
+    def forward(self, input: torch.Tensor, target: torch.Tensor) -> torch.Tensor:
         """
         Args:
             input: the shape should be BNH[WD].
             target: the shape should be BNH[WD].
-            smooth: a small constant to avoid nan.
 
         Raises:
             ValueError: When ``self.reduction`` is not one of ["mean", "sum", "none"].
@@ -129,6 +140,10 @@ def forward(self, input: torch.Tensor, target: torch.Tensor, smooth: float = 1e-
 
         # reducing only spatial dimensions (not batch nor channels)
         reduce_axis = list(range(2, len(input.shape)))
+        if self.batch:
+            # reducing spatial dimensions and batch
+            reduce_axis = [0] + reduce_axis
+
         intersection = torch.sum(target * input, dim=reduce_axis)
 
         if self.squared_pred:
@@ -143,7 +158,7 @@ def forward(self, input: torch.Tensor, target: torch.Tensor, smooth: float = 1e-
         if self.jaccard:
             denominator = 2.0 * (denominator - intersection)
 
-        f: torch.Tensor = 1.0 - (2.0 * intersection + smooth) / (denominator + smooth)
+        f: torch.Tensor = 1.0 - (2.0 * intersection + self.smooth_nr) / (denominator + self.smooth_dr)
 
         if self.reduction == LossReduction.MEAN.value:
             f = torch.mean(f)  # the batch and channel average
@@ -167,14 +182,11 @@ class MaskedDiceLoss(DiceLoss):
 
     """
 
-    def forward(
-        self, input: torch.Tensor, target: torch.Tensor, smooth: float = 1e-5, mask: Optional[torch.Tensor] = None
-    ) -> torch.Tensor:
+    def forward(self, input: torch.Tensor, target: torch.Tensor, mask: Optional[torch.Tensor] = None) -> torch.Tensor:
         """
         Args:
             input: the shape should be BNH[WD].
             target: the shape should be BNH[WD].
-            smooth: a small constant to avoid nan.
             mask: the shape should B1H[WD] or 11H[WD].
         """
         if mask is not None:
@@ -195,7 +207,7 @@ def forward(
         else:
             warnings.warn("no mask value specified for the MaskedDiceLoss.")
 
-        return super().forward(input=input, target=target, smooth=smooth)
+        return super().forward(input=input, target=target)
 
 
 class GeneralizedDiceLoss(_Loss):
@@ -218,6 +230,9 @@ def __init__(
         other_act: Optional[Callable] = None,
         w_type: Union[Weight, str] = Weight.SQUARE,
         reduction: Union[LossReduction, str] = LossReduction.MEAN,
+        smooth_nr: float = 1e-5,
+        smooth_dr: float = 1e-5,
+        batch: bool = False,
     ) -> None:
         """
         Args:
@@ -237,6 +252,10 @@ def __init__(
                 - ``"none"``: no reduction will be applied.
                 - ``"mean"``: the sum of the output will be divided by the number of elements in the output.
                 - ``"sum"``: the output will be summed.
+            smooth_nr: a small constant added to the numerator to avoid zero.
+            smooth_dr: a small constant added to the denominator to avoid nan.
+            batch: whether to sum the intersection and union areas over the batch dimension before the dividing.
+                Defaults to False, intersection over union is computed from each item in the batch.
 
         Raises:
             TypeError: When ``other_act`` is not an ``Optional[Callable]``.
@@ -262,12 +281,15 @@ def __init__(
         elif w_type == Weight.SQUARE:
             self.w_func = lambda x: torch.reciprocal(x * x)
 
-    def forward(self, input: torch.Tensor, target: torch.Tensor, smooth: float = 1e-5) -> torch.Tensor:
+        self.smooth_nr = float(smooth_nr)
+        self.smooth_dr = float(smooth_dr)
+        self.batch = batch
+
+    def forward(self, input: torch.Tensor, target: torch.Tensor) -> torch.Tensor:
         """
         Args:
             input: the shape should be BNH[WD].
             target: the shape should be BNH[WD].
-            smooth: a small constant to avoid nan.
 
         Raises:
             ValueError: When ``self.reduction`` is not one of ["mean", "sum", "none"].
@@ -305,6 +327,8 @@ def forward(self, input: torch.Tensor, target: torch.Tensor, smooth: float = 1e-
 
         # reducing only spatial dimensions (not batch nor channels)
         reduce_axis = list(range(2, len(input.shape)))
+        if self.batch:
+            reduce_axis = [0] + reduce_axis
         intersection = torch.sum(target * input, reduce_axis)
 
         ground_o = torch.sum(target, reduce_axis)
@@ -318,7 +342,9 @@ def forward(self, input: torch.Tensor, target: torch.Tensor, smooth: float = 1e-
             b[infs] = 0.0
             b[infs] = torch.max(b)
 
-        f: torch.Tensor = 1.0 - (2.0 * (intersection * w).sum(1) + smooth) / ((denominator * w).sum(1) + smooth)
+        f: torch.Tensor = 1.0 - (2.0 * (intersection * w).sum(1) + self.smooth_nr) / (
+            (denominator * w).sum(1) + self.smooth_dr
+        )
 
         if self.reduction == LossReduction.MEAN.value:
             f = torch.mean(f)  # the batch and channel average
@@ -369,7 +395,11 @@ class GeneralizedWassersteinDiceLoss(_Loss):
     """
 
     def __init__(
-        self, dist_matrix: Union[np.ndarray, torch.Tensor], reduction: Union[LossReduction, str] = LossReduction.MEAN
+        self,
+        dist_matrix: Union[np.ndarray, torch.Tensor],
+        reduction: Union[LossReduction, str] = LossReduction.MEAN,
+        smooth_nr: float = 1e-5,
+        smooth_dr: float = 1e-5,
     ) -> None:
         """
         Args:
@@ -377,6 +407,8 @@ def __init__(
                 between the classes. It must have dimension C x C where C is the
                 number of classes.
             reduction: str; reduction mode.
+            smooth_nr: a small constant added to the numerator to avoid zero.
+            smooth_dr: a small constant added to the denominator to avoid nan.
 
         Raises:
             ValueError: When ``dist_matrix`` is not a square matrix.
@@ -393,13 +425,14 @@ def __init__(
         if torch.max(self.m) != 1:
             self.m = self.m / torch.max(self.m)
         self.num_classes = self.m.size(0)
+        self.smooth_nr = float(smooth_nr)
+        self.smooth_dr = float(smooth_dr)
 
-    def forward(self, input: torch.Tensor, target: torch.Tensor, smooth: float = 1e-5) -> torch.Tensor:
+    def forward(self, input: torch.Tensor, target: torch.Tensor) -> torch.Tensor:
         """
         Args:
             input: the shape should be BNH[WD].
             target: the shape should be BNH[WD].
-            smooth: a small constant to avoid nan.
 
         """
         # Aggregate spatial dimensions
@@ -418,7 +451,7 @@ def forward(self, input: torch.Tensor, target: torch.Tensor, smooth: float = 1e-
         denom = self.compute_denominator(alpha, flat_target, wass_dist_map)
 
         # Compute and return the final loss
-        wass_dice: torch.Tensor = (2.0 * true_pos + smooth) / (denom + smooth)
+        wass_dice: torch.Tensor = (2.0 * true_pos + self.smooth_nr) / (denom + self.smooth_dr)
         wass_dice_loss: torch.Tensor = 1.0 - wass_dice
         return wass_dice_loss.mean()
 
diff --git a/monai/losses/tversky.py b/monai/losses/tversky.py
--- a/monai/losses/tversky.py
+++ b/monai/losses/tversky.py
@@ -42,6 +42,9 @@ def __init__(
         alpha: float = 0.5,
         beta: float = 0.5,
         reduction: Union[LossReduction, str] = LossReduction.MEAN,
+        smooth_nr: float = 1e-5,
+        smooth_dr: float = 1e-5,
+        batch: bool = False,
     ) -> None:
         """
         Args:
@@ -61,6 +64,12 @@ def __init__(
                 - ``"mean"``: the sum of the output will be divided by the number of elements in the output.
                 - ``"sum"``: the output will be summed.
 
+            smooth_nr: a small constant added to the numerator to avoid zero.
+            smooth_dr: a small constant added to the denominator to avoid nan.
+            batch: whether to sum the intersection and union areas over the batch dimension before the dividing.
+                Defaults to False, a Dice loss value is computed independently from each item in the batch
+                before any `reduction`.
+
         Raises:
             TypeError: When ``other_act`` is not an ``Optional[Callable]``.
             ValueError: When more than 1 of [``sigmoid=True``, ``softmax=True``, ``other_act is not None``].
@@ -80,13 +89,15 @@ def __init__(
         self.other_act = other_act
         self.alpha = alpha
         self.beta = beta
+        self.smooth_nr = float(smooth_nr)
+        self.smooth_dr = float(smooth_dr)
+        self.batch = batch
 
-    def forward(self, input: torch.Tensor, target: torch.Tensor, smooth: float = 1e-5) -> torch.Tensor:
+    def forward(self, input: torch.Tensor, target: torch.Tensor) -> torch.Tensor:
         """
         Args:
             input: the shape should be BNH[WD].
             target: the shape should be BNH[WD].
-            smooth: a small constant to avoid nan.
 
         Raises:
             ValueError: When ``self.reduction`` is not one of ["mean", "sum", "none"].
@@ -130,13 +141,15 @@ def forward(self, input: torch.Tensor, target: torch.Tensor, smooth: float = 1e-
 
         # reducing only spatial dimensions (not batch nor channels)
         reduce_axis = list(range(2, len(input.shape)))
+        if self.batch:
+            # reducing spatial dimensions and batch
+            reduce_axis = [0] + reduce_axis
 
         tp = torch.sum(p0 * g0, reduce_axis)
         fp = self.alpha * torch.sum(p0 * g1, reduce_axis)
         fn = self.beta * torch.sum(p1 * g0, reduce_axis)
-
-        numerator = tp + smooth
-        denominator = tp + fp + fn + smooth
+        numerator = tp + self.smooth_nr
+        denominator = tp + fp + fn + self.smooth_dr
 
         score: torch.Tensor = 1.0 - numerator / denominator
 


Test patch:
diff --git a/tests/test_dice_loss.py b/tests/test_dice_loss.py
--- a/tests/test_dice_loss.py
+++ b/tests/test_dice_loss.py
@@ -19,74 +19,80 @@
 
 TEST_CASES = [
     [  # shape: (1, 1, 2, 2), (1, 1, 2, 2)
-        {"include_background": True, "sigmoid": True},
+        {"include_background": True, "sigmoid": True, "smooth_nr": 1e-6, "smooth_dr": 1e-6},
         {
             "input": torch.tensor([[[[1.0, -1.0], [-1.0, 1.0]]]]),
             "target": torch.tensor([[[[1.0, 0.0], [1.0, 1.0]]]]),
-            "smooth": 1e-6,
         },
         0.307576,
     ],
     [  # shape: (2, 1, 2, 2), (2, 1, 2, 2)
-        {"include_background": True, "sigmoid": True},
+        {"include_background": True, "sigmoid": True, "smooth_nr": 1e-4, "smooth_dr": 1e-4},
         {
             "input": torch.tensor([[[[1.0, -1.0], [-1.0, 1.0]]], [[[1.0, -1.0], [-1.0, 1.0]]]]),
             "target": torch.tensor([[[[1.0, 1.0], [1.0, 1.0]]], [[[1.0, 0.0], [1.0, 0.0]]]]),
-            "smooth": 1e-4,
         },
         0.416657,
     ],
     [  # shape: (2, 2, 3), (2, 1, 3)
-        {"include_background": False, "to_onehot_y": True},
+        {"include_background": False, "to_onehot_y": True, "smooth_nr": 0, "smooth_dr": 0},
         {
             "input": torch.tensor([[[1.0, 1.0, 0.0], [0.0, 0.0, 1.0]], [[1.0, 0.0, 1.0], [0.0, 1.0, 0.0]]]),
             "target": torch.tensor([[[0.0, 0.0, 1.0]], [[0.0, 1.0, 0.0]]]),
-            "smooth": 0.0,
         },
         0.0,
     ],
     [  # shape: (2, 2, 3), (2, 1, 3)
-        {"include_background": True, "to_onehot_y": True, "sigmoid": True},
+        {"include_background": True, "to_onehot_y": True, "sigmoid": True, "smooth_nr": 1e-4, "smooth_dr": 1e-4},
         {
             "input": torch.tensor([[[-1.0, 0.0, 1.0], [1.0, 0.0, -1.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]]),
             "target": torch.tensor([[[1.0, 0.0, 0.0]], [[1.0, 1.0, 0.0]]]),
-            "smooth": 1e-4,
         },
         0.435050,
     ],
     [  # shape: (2, 2, 3), (2, 1, 3)
-        {"include_background": True, "to_onehot_y": True, "sigmoid": True, "reduction": "none"},
+        {
+            "include_background": True,
+            "to_onehot_y": True,
+            "sigmoid": True,
+            "reduction": "none",
+            "smooth_nr": 1e-4,
+            "smooth_dr": 1e-4,
+        },
         {
             "input": torch.tensor([[[-1.0, 0.0, 1.0], [1.0, 0.0, -1.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]]),
             "target": torch.tensor([[[1.0, 0.0, 0.0]], [[1.0, 1.0, 0.0]]]),
-            "smooth": 1e-4,
         },
         [[0.296529, 0.415136], [0.599976, 0.428559]],
     ],
     [  # shape: (2, 2, 3), (2, 1, 3)
-        {"include_background": True, "to_onehot_y": True, "softmax": True},
+        {"include_background": True, "to_onehot_y": True, "softmax": True, "smooth_nr": 1e-4, "smooth_dr": 1e-4},
         {
             "input": torch.tensor([[[-1.0, 0.0, 1.0], [1.0, 0.0, -1.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]]),
             "target": torch.tensor([[[1.0, 0.0, 0.0]], [[1.0, 1.0, 0.0]]]),
-            "smooth": 1e-4,
         },
         0.383713,
     ],
     [  # shape: (2, 2, 3), (2, 1, 3)
-        {"include_background": True, "to_onehot_y": True, "softmax": True, "reduction": "sum"},
+        {
+            "include_background": True,
+            "to_onehot_y": True,
+            "softmax": True,
+            "reduction": "sum",
+            "smooth_nr": 1e-4,
+            "smooth_dr": 1e-4,
+        },
         {
             "input": torch.tensor([[[-1.0, 0.0, 1.0], [1.0, 0.0, -1.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]]),
             "target": torch.tensor([[[1.0, 0.0, 0.0]], [[1.0, 1.0, 0.0]]]),
-            "smooth": 1e-4,
         },
         1.534853,
     ],
     [  # shape: (1, 1, 2, 2), (1, 1, 2, 2)
-        {"include_background": True, "sigmoid": True},
+        {"include_background": True, "sigmoid": True, "smooth_nr": 1e-6, "smooth_dr": 1e-6},
         {
             "input": torch.tensor([[[[1.0, -1.0], [-1.0, 1.0]]]]),
             "target": torch.tensor([[[[1.0, 0.0], [1.0, 1.0]]]]),
-            "smooth": 1e-6,
         },
         0.307576,
     ],
@@ -95,7 +101,6 @@
         {
             "input": torch.tensor([[[[1.0, -1.0], [-1.0, 1.0]]]]),
             "target": torch.tensor([[[[1.0, 0.0], [1.0, 1.0]]]]),
-            "smooth": 1e-5,
         },
         0.178337,
     ],
@@ -104,28 +109,55 @@
         {
             "input": torch.tensor([[[[1.0, -1.0], [-1.0, 1.0]]]]),
             "target": torch.tensor([[[[1.0, 0.0], [1.0, 1.0]]]]),
-            "smooth": 1e-5,
         },
         0.470451,
     ],
     [  # shape: (2, 1, 2, 2), (2, 1, 2, 2)
-        {"include_background": True, "other_act": torch.tanh},
+        {"include_background": True, "other_act": torch.tanh, "smooth_nr": 1e-4, "smooth_dr": 1e-4},
         {
             "input": torch.tensor([[[[1.0, -1.0], [-1.0, 1.0]]], [[[1.0, -1.0], [-1.0, 1.0]]]]),
             "target": torch.tensor([[[[1.0, 1.0], [1.0, 1.0]]], [[[1.0, 0.0], [1.0, 0.0]]]]),
-            "smooth": 1e-4,
         },
         0.999963,
     ],
     [  # shape: (2, 2, 3), (2, 1, 3)
-        {"include_background": True, "to_onehot_y": True, "other_act": lambda x: torch.log_softmax(x, dim=1)},
+        {
+            "include_background": True,
+            "to_onehot_y": True,
+            "other_act": lambda x: torch.log_softmax(x, dim=1),
+            "smooth_nr": 1e-4,
+            "smooth_dr": 1e-4,
+        },
         {
             "input": torch.tensor([[[-1.0, 0.0, 1.0], [1.0, 0.0, -1.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]]),
             "target": torch.tensor([[[1.0, 0.0, 0.0]], [[1.0, 1.0, 0.0]]]),
-            "smooth": 1e-4,
         },
         -8.522593,
     ],
+    [  # shape: (2, 1, 2, 2), (2, 1, 2, 2)
+        {"include_background": True, "other_act": torch.tanh, "smooth_nr": 1e-4, "smooth_dr": 1e-4, "batch": True},
+        {
+            "input": torch.tensor([[[[1.0, -0.0], [-1.0, 1.0]]], [[[1.0, -1.0], [-1.0, 1.0]]]]),
+            "target": torch.tensor([[[[1.0, 1.0], [1.0, 1.0]]], [[[1.0, 0.0], [1.0, 0.0]]]]),
+        },
+        0.774718,
+    ],
+    [  # shape: (2, 1, 2, 2), (2, 1, 2, 2)
+        {"include_background": True, "other_act": torch.tanh, "smooth_nr": 0, "smooth_dr": 1e-4, "batch": True},
+        {
+            "input": torch.tensor([[[[1.0, -0.0], [-1.0, 1.0]]], [[[1.0, -1.0], [-1.0, 1.0]]]]),
+            "target": torch.tensor([[[[1.0, 1.0], [1.0, 1.0]]], [[[1.0, 0.0], [1.0, 0.0]]]]),
+        },
+        0.774733,
+    ],
+    [  # shape: (2, 1, 2, 2), (2, 1, 2, 2)
+        {"include_background": True, "other_act": torch.tanh, "smooth_nr": 0, "smooth_dr": 1e-4, "batch": False},
+        {
+            "input": torch.tensor([[[[1.0, -0.0], [-1.0, 1.0]]], [[[1.0, -1.0], [-1.0, 1.0]]]]),
+            "target": torch.tensor([[[[1.0, 1.0], [1.0, 1.0]]], [[[1.0, 0.0], [1.0, 0.0]]]]),
+        },
+        0.840058,
+    ],
 ]
 
 
diff --git a/tests/test_generalized_dice_loss.py b/tests/test_generalized_dice_loss.py
--- a/tests/test_generalized_dice_loss.py
+++ b/tests/test_generalized_dice_loss.py
@@ -19,110 +19,125 @@
 
 TEST_CASES = [
     [  # shape: (1, 1, 2, 2), (1, 1, 2, 2)
-        {"include_background": True, "sigmoid": True},
+        {"include_background": True, "sigmoid": True, "smooth_nr": 1e-6, "smooth_dr": 1e-6},
         {
             "input": torch.tensor([[[[1.0, -1.0], [-1.0, 1.0]]]]),
             "target": torch.tensor([[[[1.0, 0.0], [1.0, 1.0]]]]),
-            "smooth": 1e-6,
         },
         0.307576,
     ],
     [  # shape: (2, 1, 2, 2), (2, 1, 2, 2)
-        {"include_background": True, "sigmoid": True},
+        {"include_background": True, "sigmoid": True, "smooth_nr": 1e-4, "smooth_dr": 1e-4},
         {
             "input": torch.tensor([[[[1.0, -1.0], [-1.0, 1.0]]], [[[1.0, -1.0], [-1.0, 1.0]]]]),
             "target": torch.tensor([[[[1.0, 1.0], [1.0, 1.0]]], [[[1.0, 0.0], [1.0, 0.0]]]]),
-            "smooth": 1e-4,
         },
         0.416597,
     ],
     [  # shape: (2, 2, 3), (2, 1, 3)
-        {"include_background": False, "to_onehot_y": True},
+        {"include_background": False, "to_onehot_y": True, "smooth_nr": 0.0, "smooth_dr": 0.0},
         {
             "input": torch.tensor([[[1.0, 1.0, 0.0], [0.0, 0.0, 1.0]], [[1.0, 0.0, 1.0], [0.0, 1.0, 0.0]]]),
             "target": torch.tensor([[[0.0, 0.0, 1.0]], [[0.0, 1.0, 0.0]]]),
-            "smooth": 0.0,
         },
         0.0,
     ],
     [  # shape: (2, 2, 3), (2, 1, 3)
-        {"include_background": True, "to_onehot_y": True, "sigmoid": True},
+        {"include_background": True, "to_onehot_y": True, "sigmoid": True, "smooth_nr": 1e-4, "smooth_dr": 1e-4},
         {
             "input": torch.tensor([[[-1.0, 0.0, 1.0], [1.0, 0.0, -1.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]]),
             "target": torch.tensor([[[1.0, 0.0, 0.0]], [[1.0, 1.0, 0.0]]]),
-            "smooth": 1e-4,
         },
         0.469964,
     ],
     [  # shape: (2, 2, 3), (2, 1, 3)
-        {"include_background": True, "to_onehot_y": True, "softmax": True},
+        {"include_background": True, "to_onehot_y": True, "softmax": True, "smooth_nr": 1e-4, "smooth_dr": 1e-4},
         {
             "input": torch.tensor([[[-1.0, 0.0, 1.0], [1.0, 0.0, -1.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]]),
             "target": torch.tensor([[[1.0, 0.0, 0.0]], [[1.0, 1.0, 0.0]]]),
-            "smooth": 1e-4,
         },
         0.414507,
     ],
     [  # shape: (2, 2, 3), (2, 1, 3)
-        {"include_background": True, "to_onehot_y": True, "softmax": True, "reduction": "sum"},
+        {
+            "include_background": True,
+            "to_onehot_y": True,
+            "softmax": True,
+            "reduction": "sum",
+            "smooth_nr": 1e-4,
+            "smooth_dr": 1e-4,
+        },
         {
             "input": torch.tensor([[[-1.0, 0.0, 1.0], [1.0, 0.0, -1.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]]),
             "target": torch.tensor([[[1.0, 0.0, 0.0]], [[1.0, 1.0, 0.0]]]),
-            "smooth": 1e-4,
         },
         0.829015,
     ],
     [  # shape: (2, 2, 3), (2, 1, 3)
-        {"include_background": True, "to_onehot_y": True, "softmax": True, "reduction": "none"},
+        {
+            "include_background": True,
+            "to_onehot_y": True,
+            "softmax": True,
+            "reduction": "none",
+            "smooth_nr": 1e-4,
+            "smooth_dr": 1e-4,
+        },
         {
             "input": torch.tensor([[[-1.0, 0.0, 1.0], [1.0, 0.0, -1.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]]),
             "target": torch.tensor([[[1.0, 0.0, 0.0]], [[1.0, 1.0, 0.0]]]),
-            "smooth": 1e-4,
         },
         [0.273476, 0.555539],
     ],
     [  # shape: (2, 2, 3), (2, 1, 3)
-        {"include_background": False, "to_onehot_y": True},
+        {"include_background": False, "to_onehot_y": True, "smooth_nr": 1e-8, "smooth_dr": 1e-8},
         {
             "input": torch.tensor([[[1.0, 1.0, 0.0], [0.0, 0.0, 1.0]], [[1.0, 0.0, 1.0], [0.0, 1.0, 0.0]]]),
             "target": torch.tensor([[[0.0, 0.0, 0.0]], [[0.0, 0.0, 0.0]]]),
-            "smooth": 1e-8,
         },
         0.0,
     ],
     [  # shape: (1, 1, 2, 2), (1, 1, 2, 2)
-        {"include_background": True, "sigmoid": True},
+        {"include_background": True, "sigmoid": True, "smooth_nr": 1e-6, "smooth_dr": 1e-6},
         {
             "input": torch.tensor([[[[1.0, -1.0], [-1.0, 1.0]]]]),
             "target": torch.tensor([[[[1.0, 0.0], [1.0, 1.0]]]]),
-            "smooth": 1e-6,
         },
         0.307576,
     ],
     [  # shape: (1, 2, 4), (1, 1, 4)
-        {"include_background": True, "to_onehot_y": True, "softmax": True, "w_type": "simple"},
+        {
+            "include_background": True,
+            "to_onehot_y": True,
+            "softmax": True,
+            "w_type": "simple",
+            "smooth_nr": 0,
+            "smooth_dr": 0,
+        },
         {
             "input": torch.tensor([[[0.0, 10.0, 10.0, 10.0], [10.0, 0.0, 0.0, 0.0]]]),
             "target": torch.tensor([[[1, 1, 0, 0]]]),
-            "smooth": 0.0,
         },
         0.250023,
     ],
     [  # shape: (2, 1, 2, 2), (2, 1, 2, 2)
-        {"include_background": True, "other_act": torch.tanh},
+        {"include_background": True, "other_act": torch.tanh, "smooth_nr": 1e-4, "smooth_dr": 1e-4},
         {
             "input": torch.tensor([[[[1.0, -1.0], [-1.0, 1.0]]], [[[1.0, -1.0], [-1.0, 1.0]]]]),
             "target": torch.tensor([[[[1.0, 1.0], [1.0, 1.0]]], [[[1.0, 0.0], [1.0, 0.0]]]]),
-            "smooth": 1e-4,
         },
         0.99970,
     ],
     [  # shape: (2, 2, 3), (2, 1, 3)
-        {"include_background": True, "to_onehot_y": True, "other_act": lambda x: torch.log_softmax(x, dim=1)},
+        {
+            "include_background": True,
+            "to_onehot_y": True,
+            "other_act": lambda x: torch.log_softmax(x, dim=1),
+            "smooth_nr": 1e-4,
+            "smooth_dr": 1e-4,
+        },
         {
             "input": torch.tensor([[[-1.0, 0.0, 1.0], [1.0, 0.0, -1.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]]),
             "target": torch.tensor([[[1.0, 0.0, 0.0]], [[1.0, 1.0, 0.0]]]),
-            "smooth": 1e-4,
         },
         -0.097833,
     ],
diff --git a/tests/test_masked_dice_loss.py b/tests/test_masked_dice_loss.py
--- a/tests/test_masked_dice_loss.py
+++ b/tests/test_masked_dice_loss.py
@@ -19,96 +19,100 @@
 
 TEST_CASES = [
     [  # shape: (1, 1, 2, 2), (1, 1, 2, 2)
-        {"include_background": True, "sigmoid": True},
+        {"include_background": True, "sigmoid": True, "smooth_nr": 1e-6, "smooth_dr": 1e-6},
         {
             "input": torch.tensor([[[[1.0, -1.0], [-1.0, 1.0]]]]),
             "target": torch.tensor([[[[1.0, 0.0], [1.0, 1.0]]]]),
             "mask": torch.tensor([[[[0.0, 0.0], [1.0, 1.0]]]]),
-            "smooth": 1e-6,
         },
         0.500,
     ],
     [  # shape: (2, 1, 2, 2), (2, 1, 2, 2)
-        {"include_background": True, "sigmoid": True},
+        {"include_background": True, "sigmoid": True, "smooth_nr": 1e-4, "smooth_dr": 1e-4},
         {
             "input": torch.tensor([[[[1.0, -1.0], [-1.0, 1.0]]], [[[1.0, -1.0], [-1.0, 1.0]]]]),
             "target": torch.tensor([[[[1.0, 1.0], [1.0, 1.0]]], [[[1.0, 0.0], [1.0, 0.0]]]]),
             "mask": torch.tensor([[[[1.0, 1.0], [1.0, 1.0]]], [[[1.0, 1.0], [0.0, 0.0]]]]),
-            "smooth": 1e-4,
         },
         0.422969,
     ],
     [  # shape: (2, 2, 3), (2, 1, 3)
-        {"include_background": False, "to_onehot_y": True},
+        {"include_background": False, "to_onehot_y": True, "smooth_nr": 0, "smooth_dr": 0},
         {
             "input": torch.tensor([[[1.0, 1.0, 0.0], [0.0, 0.0, 1.0]], [[1.0, 0.0, 1.0], [0.0, 1.0, 0.0]]]),
             "target": torch.tensor([[[0.0, 0.0, 1.0]], [[0.0, 1.0, 0.0]]]),
             "mask": torch.tensor([[[1.0, 1.0, 1.0]], [[0.0, 1.0, 0.0]]]),
-            "smooth": 0.0,
         },
         0.0,
     ],
     [  # shape: (2, 2, 3), (2, 1, 3)
-        {"include_background": True, "to_onehot_y": True, "sigmoid": True},
+        {"include_background": True, "to_onehot_y": True, "sigmoid": True, "smooth_nr": 1e-4, "smooth_dr": 1e-4},
         {
             "input": torch.tensor([[[-1.0, 0.0, 1.0], [1.0, 0.0, -1.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]]),
             "target": torch.tensor([[[1.0, 0.0, 0.0]], [[1.0, 1.0, 0.0]]]),
             "mask": torch.tensor([[[1.0, 1.0, 0.0]]]),
-            "smooth": 1e-4,
         },
         0.47033,
     ],
     [  # shape: (2, 2, 3), (2, 1, 3)
-        {"include_background": True, "to_onehot_y": True, "sigmoid": True, "reduction": "none"},
+        {
+            "include_background": True,
+            "to_onehot_y": True,
+            "sigmoid": True,
+            "reduction": "none",
+            "smooth_nr": 1e-4,
+            "smooth_dr": 1e-4,
+        },
         {
             "input": torch.tensor([[[-1.0, 0.0, 1.0], [1.0, 0.0, -1.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]]),
             "target": torch.tensor([[[1.0, 0.0, 0.0]], [[1.0, 1.0, 0.0]]]),
-            "smooth": 1e-4,
         },
         [[0.296529, 0.415136], [0.599976, 0.428559]],
     ],
     [  # shape: (2, 2, 3), (2, 1, 3)
-        {"include_background": True, "to_onehot_y": True, "softmax": True},
+        {"include_background": True, "to_onehot_y": True, "softmax": True, "smooth_nr": 1e-4, "smooth_dr": 1e-4},
         {
             "input": torch.tensor([[[-1.0, 0.0, 1.0], [1.0, 0.0, -1.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]]),
             "target": torch.tensor([[[1.0, 0.0, 0.0]], [[1.0, 1.0, 0.0]]]),
-            "smooth": 1e-4,
         },
         0.383713,
     ],
     [  # shape: (2, 2, 3), (2, 1, 3)
-        {"include_background": True, "to_onehot_y": True, "softmax": True, "reduction": "sum"},
+        {
+            "include_background": True,
+            "to_onehot_y": True,
+            "softmax": True,
+            "reduction": "sum",
+            "smooth_nr": 1e-4,
+            "smooth_dr": 1e-4,
+        },
         {
             "input": torch.tensor([[[-1.0, 0.0, 1.0], [1.0, 0.0, -1.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]]),
             "target": torch.tensor([[[1.0, 0.0, 0.0]], [[1.0, 1.0, 0.0]]]),
-            "smooth": 1e-4,
         },
         1.534853,
     ],
     [  # shape: (1, 1, 2, 2), (1, 1, 2, 2)
-        {"include_background": True, "sigmoid": True},
+        {"include_background": True, "sigmoid": True, "smooth_nr": 1e-6, "smooth_dr": 1e-6},
         {
             "input": torch.tensor([[[[1.0, -1.0], [-1.0, 1.0]]]]),
             "target": torch.tensor([[[[1.0, 0.0], [1.0, 1.0]]]]),
-            "smooth": 1e-6,
         },
         0.307576,
     ],
     [  # shape: (1, 1, 2, 2), (1, 1, 2, 2)
-        {"include_background": True, "sigmoid": True, "squared_pred": True},
+        {"include_background": True, "sigmoid": True, "squared_pred": True, "smooth_nr": 1e-5, "smooth_dr": 1e-5},
         {
             "input": torch.tensor([[[[1.0, -1.0], [-1.0, 1.0]]]]),
             "target": torch.tensor([[[[1.0, 0.0], [1.0, 1.0]]]]),
-            "smooth": 1e-5,
         },
         0.178337,
     ],
     [  # shape: (1, 1, 2, 2), (1, 1, 2, 2)
-        {"include_background": True, "sigmoid": True, "jaccard": True},
+        {"include_background": True, "sigmoid": True, "jaccard": True, "smooth_nr": 1e-5, "smooth_dr": 1e-5},
         {
             "input": torch.tensor([[[[1.0, -1.0], [-1.0, 1.0]]]]),
             "target": torch.tensor([[[[1.0, 0.0], [1.0, 1.0]]]]),
-            "smooth": 1e-5,
         },
         0.470451,
     ],
diff --git a/tests/test_seg_loss_integration.py b/tests/test_seg_loss_integration.py
--- a/tests/test_seg_loss_integration.py
+++ b/tests/test_seg_loss_integration.py
@@ -20,7 +20,9 @@
 from monai.losses import DiceLoss, FocalLoss, GeneralizedDiceLoss, TverskyLoss
 
 TEST_CASES = [
-    [DiceLoss, {"to_onehot_y": True, "squared_pred": True}, {"smooth": 1e-4}],
+    [DiceLoss, {"to_onehot_y": True, "squared_pred": True, "smooth_nr": 1e-4, "smooth_dr": 1e-4}, {}],
+    [DiceLoss, {"to_onehot_y": True, "squared_pred": True, "smooth_nr": 0, "smooth_dr": 1e-3}, {}],
+    [DiceLoss, {"to_onehot_y": True, "squared_pred": True, "batch": True}, {}],
     [DiceLoss, {"to_onehot_y": True, "sigmoid": True}, {}],
     [DiceLoss, {"to_onehot_y": True, "softmax": True}, {}],
     [FocalLoss, {"gamma": 1.5, "weight": torch.tensor([1, 2])}, {}],
@@ -30,6 +32,7 @@
     [GeneralizedDiceLoss, {"to_onehot_y": True, "sigmoid": True, "w_type": "simple"}, {}],
     [GeneralizedDiceLoss, {"to_onehot_y": True, "sigmoid": True, "w_type": "uniform"}, {}],
     [TverskyLoss, {"to_onehot_y": True, "softmax": True, "alpha": 0.8, "beta": 0.2}, {}],
+    [TverskyLoss, {"to_onehot_y": True, "softmax": True, "alpha": 0.8, "beta": 0.2, "batch": True}, {}],
     [TverskyLoss, {"to_onehot_y": True, "softmax": True, "alpha": 1.0, "beta": 0.0}, {}],
 ]
 
diff --git a/tests/test_tversky_loss.py b/tests/test_tversky_loss.py
--- a/tests/test_tversky_loss.py
+++ b/tests/test_tversky_loss.py
@@ -19,101 +19,134 @@
 
 TEST_CASES = [
     [  # shape: (1, 1, 2, 2), (1, 1, 2, 2)
-        {"include_background": True, "sigmoid": True},
+        {"include_background": True, "sigmoid": True, "smooth_nr": 1e-6, "smooth_dr": 1e-6},
         {
             "input": torch.tensor([[[[1.0, -1.0], [-1.0, 1.0]]]]),
             "target": torch.tensor([[[[1.0, 0.0], [1.0, 1.0]]]]),
-            "smooth": 1e-6,
         },
         0.307576,
     ],
     [  # shape: (2, 1, 2, 2), (2, 1, 2, 2)
-        {"include_background": True, "sigmoid": True},
+        {"include_background": True, "sigmoid": True, "smooth_nr": 1e-4, "smooth_dr": 1e-4},
         {
             "input": torch.tensor([[[[1.0, -1.0], [-1.0, 1.0]]], [[[1.0, -1.0], [-1.0, 1.0]]]]),
             "target": torch.tensor([[[[1.0, 1.0], [1.0, 1.0]]], [[[1.0, 0.0], [1.0, 0.0]]]]),
-            "smooth": 1e-4,
         },
         0.416657,
     ],
     [  # shape: (2, 2, 3), (2, 1, 3)
-        {"include_background": False, "to_onehot_y": True},
+        {"include_background": False, "to_onehot_y": True, "smooth_nr": 0, "smooth_dr": 0},
         {
             "input": torch.tensor([[[1.0, 1.0, 0.0], [0.0, 0.0, 1.0]], [[1.0, 0.0, 1.0], [0.0, 1.0, 0.0]]]),
             "target": torch.tensor([[[0.0, 0.0, 1.0]], [[0.0, 1.0, 0.0]]]),
-            "smooth": 0.0,
         },
         0.0,
     ],
     [  # shape: (2, 2, 3), (2, 1, 3)
-        {"include_background": True, "to_onehot_y": True, "sigmoid": True},
+        {"include_background": False, "to_onehot_y": True, "smooth_nr": 0, "smooth_dr": 1e-3},
+        {
+            "input": torch.tensor([[[1.0, 1.0, 0.0], [0.0, 0.0, 1.0]], [[1.0, 0.0, 1.0], [0.0, 1.0, 0.0]]]),
+            "target": torch.tensor([[[0.0, 0.0, 1.0]], [[0.0, 1.0, 0.0]]]),
+        },
+        0.000999,
+    ],
+    [  # shape: (2, 2, 3), (2, 1, 3)
+        {"include_background": True, "to_onehot_y": True, "sigmoid": True, "smooth_nr": 1e-4, "smooth_dr": 1e-4},
         {
             "input": torch.tensor([[[-1.0, 0.0, 1.0], [1.0, 0.0, -1.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]]),
             "target": torch.tensor([[[1.0, 0.0, 0.0]], [[1.0, 1.0, 0.0]]]),
-            "smooth": 1e-4,
         },
         0.435050,
     ],
     [  # shape: (2, 2, 3), (2, 1, 3)
-        {"include_background": True, "to_onehot_y": True, "sigmoid": True, "reduction": "sum"},
+        {"include_background": True, "to_onehot_y": True, "sigmoid": True, "batch": True},
+        {
+            "input": torch.tensor([[[-1.0, 0.0, 1.0], [1.0, 0.0, -1.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]]),
+            "target": torch.tensor([[[1.0, 0.0, 0.0]], [[1.0, 1.0, 0.0]]]),
+        },
+        0.422979,
+    ],
+    [  # shape: (2, 2, 3), (2, 1, 3)
+        {
+            "include_background": True,
+            "to_onehot_y": True,
+            "sigmoid": True,
+            "reduction": "sum",
+            "smooth_nr": 1e-4,
+            "smooth_dr": 1e-4,
+        },
         {
             "input": torch.tensor([[[-1.0, 0.0, 1.0], [1.0, 0.0, -1.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]]),
             "target": torch.tensor([[[1.0, 0.0, 0.0]], [[1.0, 1.0, 0.0]]]),
-            "smooth": 1e-4,
         },
         1.74013,
     ],
     [  # shape: (2, 2, 3), (2, 1, 3)
-        {"include_background": True, "to_onehot_y": True, "softmax": True},
+        {"include_background": True, "to_onehot_y": True, "softmax": True, "smooth_nr": 1e-4, "smooth_dr": 1e-4},
         {
             "input": torch.tensor([[[-1.0, 0.0, 1.0], [1.0, 0.0, -1.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]]),
             "target": torch.tensor([[[1.0, 0.0, 0.0]], [[1.0, 1.0, 0.0]]]),
-            "smooth": 1e-4,
         },
         0.383713,
     ],
     [  # shape: (2, 2, 3), (2, 1, 3)
-        {"include_background": True, "to_onehot_y": True, "softmax": True, "reduction": "none"},
+        {
+            "include_background": True,
+            "to_onehot_y": True,
+            "softmax": True,
+            "reduction": "none",
+            "smooth_nr": 1e-4,
+            "smooth_dr": 1e-4,
+        },
         {
             "input": torch.tensor([[[-1.0, 0.0, 1.0], [1.0, 0.0, -1.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]]),
             "target": torch.tensor([[[1.0, 0.0, 0.0]], [[1.0, 1.0, 0.0]]]),
-            "smooth": 1e-4,
         },
         [[0.210961, 0.295339], [0.599952, 0.428547]],
     ],
     [  # shape: (1, 1, 2, 2), (1, 1, 2, 2)
-        {"include_background": True, "sigmoid": True, "alpha": 0.3, "beta": 0.7},
+        {"include_background": True, "sigmoid": True, "alpha": 0.3, "beta": 0.7, "smooth_nr": 1e-6, "smooth_dr": 1e-6},
         {
             "input": torch.tensor([[[[1.0, -1.0], [-1.0, 1.0]]]]),
             "target": torch.tensor([[[[1.0, 0.0], [1.0, 1.0]]]]),
-            "smooth": 1e-6,
         },
         0.3589,
     ],
     [  # shape: (1, 1, 2, 2), (1, 1, 2, 2)
-        {"include_background": True, "sigmoid": True, "alpha": 0.7, "beta": 0.3},
+        {"include_background": True, "sigmoid": True, "alpha": 0.7, "beta": 0.3, "smooth_nr": 1e-6, "smooth_dr": 1e-6},
         {
             "input": torch.tensor([[[[1.0, -1.0], [-1.0, 1.0]]]]),
             "target": torch.tensor([[[[1.0, 0.0], [1.0, 1.0]]]]),
-            "smooth": 1e-6,
         },
         0.247366,
     ],
     [  # shape: (2, 1, 2, 2), (2, 1, 2, 2)
-        {"include_background": True, "other_act": torch.tanh},
+        {"include_background": True, "other_act": torch.tanh, "smooth_nr": 1e-4, "smooth_dr": 1e-4},
+        {
+            "input": torch.tensor([[[[1.0, -1.0], [-1.0, 1.0]]], [[[1.0, -1.0], [-1.0, 1.0]]]]),
+            "target": torch.tensor([[[[1.0, 1.0], [1.0, 1.0]]], [[[1.0, 0.0], [1.0, 0.0]]]]),
+        },
+        0.999963,
+    ],
+    [  # shape: (2, 1, 2, 2), (2, 1, 2, 2)
+        {"include_background": True, "other_act": torch.tanh, "smooth_nr": 0, "smooth_dr": 1e-3, "batch": True},
         {
             "input": torch.tensor([[[[1.0, -1.0], [-1.0, 1.0]]], [[[1.0, -1.0], [-1.0, 1.0]]]]),
             "target": torch.tensor([[[[1.0, 1.0], [1.0, 1.0]]], [[[1.0, 0.0], [1.0, 0.0]]]]),
-            "smooth": 1e-4,
         },
         0.999963,
     ],
     [  # shape: (2, 2, 3), (2, 1, 3)
-        {"include_background": True, "to_onehot_y": True, "other_act": lambda x: torch.log_softmax(x, dim=1)},
+        {
+            "include_background": True,
+            "to_onehot_y": True,
+            "other_act": lambda x: torch.log_softmax(x, dim=1),
+            "smooth_nr": 1e-4,
+            "smooth_dr": 1e-4,
+        },
         {
             "input": torch.tensor([[[-1.0, 0.0, 1.0], [1.0, 0.0, -1.0]], [[0.0, 0.0, 0.0], [0.0, 0.0, 0.0]]]),
             "target": torch.tensor([[[1.0, 0.0, 0.0]], [[1.0, 1.0, 0.0]]]),
-            "smooth": 1e-4,
         },
         -8.533317,
     ],