Repository: Project-MONAI/MONAI

Issue:
Deconvolutions improve performance of AH-Net
**Describe the bug**
In the current environment deconvolutions work significantly faster with mixed precision than trilinear interpolation for upsampling. In the current Monai implementation of AH-Net the `upsample_mode` defaults to "trilinear", which is suboptimal.  

**To Reproduce**
Run training with `upsample_mode="trilinear"` and `upsample_mode="transpose"`.

**Expected behavior**
The default `upsample_mode` is changed to `transpose`.

**Screenshots**
Throughput measured on V100 and A100 for a 3D workload.
![image](https://user-images.githubusercontent.com/43240942/92876049-f75a6180-f409-11ea-9bdf-0851fd7dca40.png)


**Environment (please complete the following information):**
 - Ubuntu 18.04
 - Python 3.6
 - MONAI version f1998b72a941d1e5f9578a66dc1c20b01913caab
 - CUDA 11, cuDNN 8.0.3/8.0.4
 - GPUs: V100, A100

**Additional context**
In cuDNN 8.0.3 deconvolutions have Tensor Core support which improves their performance significantly. 
Currently there are issues being tracked for Pytorch and cuDNN which might improve performance. I'll monitor the status and update if needed.



Discussion:
Hi @yiheng-wang-nv ,

I remember you fixed some determinism issue about the upsampling in AHNet, is it related to this ticket?

Thanks.

> Hi @yiheng-wang-nv ,
> 
> I remember you fixed some determinism issue about the upsampling in AHNet, is it related to this ticket?
> 
> Thanks.

that's noted in the docstring @Nic-Ma 
https://github.com/Project-MONAI/MONAI/blob/8fc4f5dd5ea4d3726fdd9e001eacf517263625ac/monai/networks/nets/ahnet.py#L353-L355
> Hi @yiheng-wang-nv ,
> 
> I remember you fixed some determinism issue about the upsampling in AHNet, is it related to this ticket?
> 
> Thanks.

FYI, the non-determinism issue is explained in an old [PR](https://github.com/Project-MONAI/MONAI/pull/815). Generally speaking, `torch.nn.functional.interpolate() ` and `torch.nn.MaxPool1d/2d/3d()`  (if kernel size != stride) use `atomicAdd ` which introduces the non-determinism results. 
This ticket is about the comparisons for computing speed between `trilinear` and `deconvolution`, and since our `deconvolution` mode ensures the determinism, the kernel size for `maxpool` layer has been changed (shown below), will it have an impact on the speed?
```
if upsample_mode == "transpose":
    self.maxpool = pool_type(kernel_size=(2, 2, 2)[-spatial_dims:], stride=2)
else:
    self.maxpool = pool_type(kernel_size=(3, 3, 3)[-spatial_dims:], stride=2, padding=1)

```

Resolving patch:
diff --git a/monai/networks/nets/ahnet.py b/monai/networks/nets/ahnet.py
--- a/monai/networks/nets/ahnet.py
+++ b/monai/networks/nets/ahnet.py
@@ -332,12 +332,17 @@ class AHNet(nn.Module):
     """
     AHNet based on `Anisotropic Hybrid Network <https://arxiv.org/pdf/1711.08580.pdf>`_.
     Adapted from `lsqshr's official code <https://github.com/lsqshr/AH-Net/blob/master/net3d.py>`_.
-    The model supports 2D or 3D inputs, as for the original 3D version.
+    Except from the original network that supports 3D inputs, this implementation also supports 2D inputs.
+    According to the `tests for deconvolutions <https://github.com/Project-MONAI/MONAI/issues/1023>`_, using
+    ``"transpose"`` rather than linear interpolations is faster. Therefore, this implementation sets ``"transpose"``
+    as the default upsampling method.
+
     To meet to requirements of the structure, the input size of the first ``dim-1`` dimensions should be divisible
-    by 32 and no less than 128. If you need to use lower sizes, please reduce the largest blocks in PSP
-    module and change the ``num_input_features`` in Final module.
-    In addition, to utilize the "transpose" upsample mode, please ensure that the input size of the first ``dim-1`` dimensions
-    should be divisible by 128. In order to use pretrained weights from 2D FCN/MCFCN, please call the `copy_from` function,
+    by 128. In addition, for linear interpolation based upsampling modes, the input size of the first ``dim-1``
+    dimensions should be divisible by 32 and no less than 128. If you need to use lower sizes, please reduce the
+    largest blocks in PSP module and change the ``num_input_features`` in Final module.
+
+    In order to use pretrained weights from 2D FCN/MCFCN, please call the `copy_from` function,
     for example:
 
     .. code-block:: python
@@ -352,7 +357,7 @@ class AHNet(nn.Module):
         out_channels: number of output channels for the network. Defaults to 1.
         upsample_mode: [``"transpose"``, ``"bilinear"``, ``"trilinear"``]
             The mode of upsampling manipulations.
-            Using the last two modes cannot guarantee the model's reproducibility. Defaults to ``trilinear``.
+            Using the last two modes cannot guarantee the model's reproducibility. Defaults to ``transpose``.
 
             - ``"transpose"``, uses transposed convolution layers.
             - ``"bilinear"``, uses bilinear interpolate.
@@ -364,7 +369,7 @@ def __init__(
         layers: tuple = (3, 4, 6, 3),
         spatial_dims: int = 3,
         out_channels: int = 1,
-        upsample_mode: str = "trilinear",
+        upsample_mode: str = "transpose",
     ):
         self.inplanes = 64
         super(AHNet, self).__init__()


Test patch: