Repository: Project-MONAI/MONAI

Issue:
`Pad` uses numpy though pytorch backend is available
**Describe the bug**
According to the doc string of `Pad`:
https://github.com/Project-MONAI/MONAI/blob/f519699f4b709aa69f23c252842e4798cac293d8/monai/transforms/croppad/array.py#L88-L89
However, sometimes `Pad` uses numpy though pytorch backend is available.

**To Reproduce**
Run the following script, which wraps pad of numpy and torch by printing corresponding information first, and pads the input with Monai's transform to see the output:
```python
import torch

from monai.transforms import Pad
from monai.transforms.croppad import functional as F
from monai.utils import PytorchPadMode

class Wrap:
    def __init__(self, func, info: str):
        self.func = func
        self.info = info

    def __call__(self, *args, **kwargs):
        print(self.info)
        return self.func(*args, **kwargs)

if __name__ == '__main__':
    F._np_pad = Wrap(F._np_pad, 'np')
    F._pt_pad = Wrap(F._pt_pad, 'pt')
    pad = Pad(mode=PytorchPadMode.CONSTANT)
    x = torch.randn(1, 5, 5)
    print(pad(x, [(0, 1), (1, 1), (0, 0)]).shape)
```

**Expected behavior**

PyTorch is available in the script, so it should output `pt` (indicating that PyTorch pad is used)

**Actual behavior**
The script outputs `np` (indicating that numpy pad is used).

**Environment**

```
================================
Printing MONAI config...
================================
MONAI version: 1.1.0+88.gf519699f
Numpy version: 1.24.2
Pytorch version: 1.13.1.post200
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: f519699f4b709aa69f23c252842e4798cac293d8

Optional dependencies:
Pytorch Ignite version: 0.4.10
ITK version: 5.3.0
Nibabel version: 5.0.0
scikit-image version: 0.19.3
Pillow version: 9.2.0
Tensorboard version: 2.12.0
gdown version: 4.6.2
TorchVision version: 0.14.1a0+b69fce3
tqdm version: 4.64.1
lmdb version: 1.4.0
psutil version: 5.9.4
pandas version: 1.5.3
einops version: 0.6.0
transformers version: 4.21.3
mlflow version: 2.1.1
pynrrd version: 1.0.0

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies


================================
Printing system config...
================================
System: Linux
Linux version: Ubuntu 22.04.2 LTS
Platform: Linux-5.19.0-32-generic-x86_64-with-glibc2.35
Processor: x86_64
Machine: x86_64
Python version: 3.10.9
Process name: python
Command: ['python', '-c', 'import monai; monai.config.print_debug_info()']
Open files: []
Num physical CPUs: 8
Num logical CPUs: 16
Num usable CPUs: 16
CPU usage (%): [2.0, 2.7, 5.3, 3.4, 4.1, 2.1, 2.7, 0.0, 3.4, 4.2, 7.2, 12.2, 4.8, 6.2, 2.1, 97.9]
CPU freq. (MHz): 3733
Load avg. in last 1, 5, 15 mins (%): [2.6, 2.4, 1.9]
Disk usage (%): 47.3
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 62.7
Available memory (GB): 32.0
Used memory (GB): 29.7

================================
Printing GPU config...
================================
Num GPUs: 1
Has CUDA: True
CUDA version: 11.2
cuDNN enabled: True
cuDNN version: 8401
Current device: 0
Library compiled for CUDA architectures: ['sm_35', 'sm_50', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'compute_86']
GPU 0 Name: NVIDIA GeForce GTX 1080 Ti
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 28
GPU 0 Total memory (GB): 10.9
GPU 0 CUDA capability (maj.min): 6.1
```

**Additional context**

I think the condition for PyTorch here is too strict.

https://github.com/Project-MONAI/MONAI/blob/f519699f4b709aa69f23c252842e4798cac293d8/monai/transforms/croppad/functional.py#L70-L74



Discussion:
thanks for reporting, would you like to help enhance the logic? the main testing cases are listed here https://github.com/Project-MONAI/MONAI/blob/dev/tests/test_pad_mode.py
Hi @wyli, I'd like to, but maybe two weeks later, since only then will I have enough time. 

https://github.com/Project-MONAI/MONAI/pull/4749 introduced this behavior for fixing another issue. Could you please give some hints on how to keep these compatible?
I see, according to https://github.com/pytorch/pytorch/issues/40763#issue-648018930, `torch.nn.functional.pad` does not support input tensors with dtype other than float32 or float64 for modes "replicate" and "reflect". The fix will be simple, change this condition:
https://github.com/Project-MONAI/MONAI/blob/f519699f4b709aa69f23c252842e4798cac293d8/monai/transforms/croppad/functional.py#L70-L74
to:
```python
 _pad = ( 
     _np_pad 
     if mode in {"reflect", "replicate"} and img.dtype in {torch.int16, torch.int64, torch.bool, torch.uint8} 
     else  _pt_pad
 ) 
```
Or, we can just use PyTorch pad here first and use NumPy pad if any exception occurs. I'm not sure which one is desired.
ok, I remember the _pt_pad was not very robust, it's only compatible with a few combinations of dtypes + padding modes. I can have a look soon.

Resolving patch:
diff --git a/monai/transforms/croppad/functional.py b/monai/transforms/croppad/functional.py
--- a/monai/transforms/croppad/functional.py
+++ b/monai/transforms/croppad/functional.py
@@ -15,6 +15,8 @@
 
 from __future__ import annotations
 
+import warnings
+
 import numpy as np
 import torch
 from torch.nn.functional import pad as pad_pt
@@ -29,7 +31,12 @@
 
 
 def _np_pad(img: torch.Tensor, pad_width: list[tuple[int, int]], mode: str, **kwargs) -> torch.Tensor:
-    img_np = img.detach().cpu().numpy() if isinstance(img, torch.Tensor) else img
+    if isinstance(img, torch.Tensor):
+        if img.is_cuda:
+            warnings.warn(f"Padding: moving img {img.shape} from cuda to cpu for dtype={img.dtype} mode={mode}.")
+        img_np = img.detach().cpu().numpy()
+    else:
+        img_np = img
     mode = convert_pad_mode(dst=img_np, mode=mode).value
     if mode == "constant" and "value" in kwargs:
         kwargs["constant_values"] = kwargs.pop("value")
@@ -40,9 +47,15 @@ def _np_pad(img: torch.Tensor, pad_width: list[tuple[int, int]], mode: str, **kw
 
 
 def _pt_pad(img: torch.Tensor, pad_width: list[tuple[int, int]], mode: str, **kwargs) -> torch.Tensor:
+    mode = convert_pad_mode(dst=img, mode=mode).value
+    if mode == "constant" and "constant_values" in kwargs:
+        _kwargs = kwargs.copy()
+        _kwargs["value"] = _kwargs.pop("constant_values")
+    else:
+        _kwargs = kwargs
     pt_pad_width = [val for sublist in pad_width[1:] for val in sublist[::-1]][::-1]
     # torch.pad expects `[B, C, H, W, [D]]` shape
-    return pad_pt(img.unsqueeze(0), pt_pad_width, mode=mode, **kwargs).squeeze(0)
+    return pad_pt(img.unsqueeze(0), pt_pad_width, mode=mode, **_kwargs).squeeze(0)
 
 
 def pad_nd(img: torch.Tensor, to_pad: list[tuple[int, int]], mode: str, **kwargs):
@@ -68,14 +81,14 @@ def pad_nd(img: torch.Tensor, to_pad: list[tuple[int, int]], mode: str, **kwargs
     mode = convert_pad_mode(dst=img, mode=mode).value
     try:
         _pad = (
-            _pt_pad
-            if mode in {"reflect", "replicate"} and img.dtype not in {torch.int16, torch.int64, torch.bool, torch.uint8}
-            else _np_pad
+            _np_pad
+            if mode in {"reflect", "replicate"} and img.dtype in {torch.int16, torch.int64, torch.bool, torch.uint8}
+            else _pt_pad
         )
         return _pad(img, pad_width=to_pad, mode=mode, **kwargs)
     except (ValueError, TypeError, RuntimeError) as err:
         if isinstance(err, NotImplementedError) or any(
-            k in str(err) for k in ("supported", "unexpected keyword", "implemented")
+            k in str(err) for k in ("supported", "unexpected keyword", "implemented", "value")
         ):
             return _np_pad(img, pad_width=to_pad, mode=mode, **kwargs)
         raise ValueError(f"{img.shape} {to_pad} {mode} {kwargs} {img.dtype} {img.device}") from err
diff --git a/monai/transforms/utils.py b/monai/transforms/utils.py
--- a/monai/transforms/utils.py
+++ b/monai/transforms/utils.py
@@ -1628,13 +1628,13 @@ def convert_pad_mode(dst: NdarrayOrTensor, mode: str | None):
     if isinstance(dst, torch.Tensor):
         if mode == "wrap":
             mode = "circular"
-        if mode == "edge":
+        elif mode == "edge":
             mode = "replicate"
         return look_up_option(mode, PytorchPadMode)
     if isinstance(dst, np.ndarray):
         if mode == "circular":
             mode = "wrap"
-        if mode == "replicate":
+        elif mode == "replicate":
             mode = "edge"
         return look_up_option(mode, NumpyPadMode)
     raise ValueError(f"unsupported data type: {type(dst)}.")


Test patch: