RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
Labels: module: crash, module: cudnn, module: cuda, triaged, module: ddp

### 🐛 Describe the bug

I have run train.py in YOLOX project, I got the error message as title description.

when i train the script by `python tools/train.py -f exps/example/yolox_voc/yolox_voc_s.py -d 0 -b 8 --fp16 -o -c yolox_s.pth`, I meet the problem:

### To Reproduce
My sample code as the link: https://github.com/Megvii-BaseDetection/YOLOX/blob/main/tools/train.py

The error message as below:
### Error message:

2022-06-06 16:24:43 | ERROR | yolox.core.launch:98 - An error has been caught in function 'launch', process 'MainProcess' (7800), thread 'MainThread' (7924):
Traceback (most recent call last):

File "tools\train.py", line 140, in
args=(exp, args),
│ └ Namespace(batch_size=4, cache=False, ckpt='yolox_s.pth', devices=0, dist_backend='nccl', dist_url=None, exp_file='exps/exampl...
└ ╒═══════════════════╤════════════════════════════════════════════════════════════════════════════════════════════════════════...

File "d:\e\spore\yolox\yolox\core\launch.py", line 98, in launch
main_func(*args)
│ └ (╒═══════════════════╤═══════════════════════════════════════════════════════════════════════════════════════════════════════...
└ <function main at 0x0000027189606378>

File "tools\train.py", line 117, in main
trainer.train()
│ └ <function Trainer.train at 0x0000027189A686A8>
└ <yolox.core.trainer.Trainer object at 0x0000027189A74828>

File "d:\e\spore\yolox\yolox\core\trainer.py", line 76, in train
self.train_in_epoch()
│ └ <function Trainer.train_in_epoch at 0x0000027189A68D90>
└ <yolox.core.trainer.Trainer object at 0x0000027189A74828>

File "d:\e\spore\yolox\yolox\core\trainer.py", line 85, in train_in_epoch
self.train_in_iter()
│ └ <function Trainer.train_in_iter at 0x0000027189A68E18>
└ <yolox.core.trainer.Trainer object at 0x0000027189A74828>

File "d:\e\spore\yolox\yolox\core\trainer.py", line 91, in train_in_iter
self.train_one_iter()
│ └ <function Trainer.train_one_iter at 0x0000027189A68EA0>
└ <yolox.core.trainer.Trainer object at 0x0000027189A74828>

File "d:\e\spore\yolox\yolox\core\trainer.py", line 110, in train_one_iter
self.scaler.scale(loss).backward()
│ │ │ └ tensor(10.1536, device='cuda:0', grad_fn=)
│ │ └ <function GradScaler.scale at 0x00000271FE5FED90>
│ └ <torch.cuda.amp.grad_scaler.GradScaler object at 0x0000027189A74860>
└ <yolox.core.trainer.Trainer object at 0x0000027189A74828>

File "D:\F\Anaconda3\envs\yolox\lib\site-packages\torch_tensor.py", line 307, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
│ │ │ │ │ │ │ └ None
│ │ │ │ │ │ └ False
│ │ │ │ │ └ None
│ │ │ │ └ None
│ │ │ └ tensor([665425.], device='cuda:0', grad_fn=)
│ │ └ <function backward at 0x00000271FEACE1E0>
│ └ <module 'torch.autograd' from 'D:\F\Anaconda3\envs\yolox\lib\site-packages\torch\autograd\init.py'>
└ <module 'torch' from 'D:\F\Anaconda3\envs\yolox\lib\site-packages\torch\init.py'>

File "D:\F\Anaconda3\envs\yolox\lib\site-packages\torch\autograd_init_.py", line 156, in backward
allow_unreachable=True, accumulate_grad=True) # allow_unreachable flag

RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = True
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([4, 128, 20, 20], dtype=torch.half, device='cuda', requires_grad=True)
net = torch.nn.Conv2d(128, 1, kernel_size=[1, 1], padding=[0, 0], stride=[1, 1], dilation=[1, 1], groups=1)
net = net.cuda().half()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()

ConvolutionParams
data_type = CUDNN_DATA_HALF
padding = [0, 0, 0]
stride = [1, 1, 0]
dilation = [1, 1, 0]
groups = 1
deterministic = false
allow_tf32 = true
input: TensorDescriptor 00000274B9F17FB0
type = CUDNN_DATA_HALF
nbDims = 4
dimA = 4, 128, 20, 20,
strideA = 51200, 400, 20, 1,
output: TensorDescriptor 00000274B9F16F80
type = CUDNN_DATA_HALF
nbDims = 4
dimA = 4, 1, 20, 20,
strideA = 400, 400, 20, 1,
weight: FilterDescriptor 000002747033C530
type = CUDNN_DATA_HALF
tensor_format = CUDNN_TENSOR_NCHW
nbDims = 4
dimA = 1, 128, 1, 1,
Pointer addresses:
input: 00000009550AC000
output: 0000000716DFB000
weight: 00000007167FFA00
### My Action:
I have already run the code snippet as below which provided by error message. It's running successfully. No error.

import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = True
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([4, 128, 20, 20], dtype=torch.half, device='cuda', requires_grad=True)
net = torch.nn.Conv2d(128, 1, kernel_size=[1, 1], padding=[0, 0], stride=[1, 1], dilation=[1, 1], groups=1)
net = net.cuda().half()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()

Looking for your reply.
Thanks a lot.

### Versions

Collecting environment information...
PyTorch version: 1.10.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 ▒▒ͥ▒▒▒İ▒
GCC version: Could not collect
Clang version: Could not collect
CMake version: version 3.20.3
Libc version: N/A

Python version: 3.7.0 (default, Jun 28 2018, 08:04:48) [MSC v.1912 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19041-SP0
Is CUDA available: True
CUDA runtime version: 11.1.74
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3080
Nvidia driver version: 472.39
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\cudnn_ops_train64_8.dll
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.21.6
[pip3] torch==1.10.0
[pip3] torchaudio==0.10.0
[pip3] torchfile==0.1.0
[pip3] torchinfo==1.6.3
[pip3] torchnet==0.0.4
[pip3] torchvision==0.11.1
[conda] blas                      1.0                         mkl    https://repo.anaconda.com/pkgs/main
[conda] cudatoolkit               11.3.1               h59b6b97_2    https://repo.anaconda.com/pkgs/main
[conda] mkl                       2021.4.0           haa95532_640    https://repo.anaconda.com/pkgs/main
[conda] mkl-service               2.4.0            py37h2bbff1b_0    https://repo.anaconda.com/pkgs/main
[conda] mkl_fft                   1.3.1            py37h277e83a_0    https://repo.anaconda.com/pkgs/main
[conda] mkl_random                1.2.2            py37hf11a4ad_0    https://repo.anaconda.com/pkgs/main
[conda] numpy                     1.21.6                   pypi_0    pypi
[conda] pytorch                   1.10.0          py3.7_cuda11.3_cudnn8_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchaudio                0.10.0               py37_cu113    pytorch
[conda] torchfile                 0.1.0                    pypi_0    pypi
[conda] torchinfo                 1.6.3                    pypi_0    pypi
[conda] torchnet                  0.0.4                    pypi_0    pypi
[conda] torchvision               0.11.1               py37_cu113    pytorch
(yolo)


yolo is my virtual environment.

**My computer's environment set is:**
Collecting environment information...
PyTorch version: 1.7.1
Is debug build: False
CUDA used to build PyTorch: 11.0
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 ▒▒ͥ▒▒▒İ▒
GCC version: Could not collect
Clang version: Could not collect
CMake version: version 3.20.3
Libc version: N/A

Python version: 3.7.4 (default, Aug  9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19041-SP0
Is CUDA available: True
CUDA runtime version: 11.1.74
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3080
Nvidia driver version: 472.39
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\cudnn_ops_train64_8.dll
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: False

Versions of relevant libraries:
[pip3] numpy==1.16.5
[pip3] numpydoc==0.9.1
[pip3] torch==1.7.1
[conda] blas                      1.0                         mkl    https://repo.anaconda.com/pkgs/main
[conda] cudatoolkit               11.0.221             h74a9793_0    https://repo.anaconda.com/pkgs/main
[conda] mkl                       2019.4                      245    https://repo.anaconda.com/pkgs/main
[conda] mkl-service               2.3.0            py37hb782905_0    https://repo.anaconda.com/pkgs/main
[conda] mkl_fft                   1.0.14           py37h14836fe_0    https://repo.anaconda.com/pkgs/main
[conda] mkl_random                1.1.0            py37h675688f_0    https://repo.anaconda.com/pkgs/main
[conda] numpy                     1.16.5           py37h19fb1c0_0    https://repo.anaconda.com/pkgs/main
[conda] numpy-base                1.16.5           py37hc3f5095_0    https://repo.anaconda.com/pkgs/main
[conda] numpydoc                  0.9.1                      py_0    https://repo.anaconda.com/pkgs/main
[conda] pytorch                   1.7.1           py3.7_cuda110_cudnn8_0    pytorch

I'm not sure virtual environment cuda version and computer cuda version are influential each other.

cc @csarofeen @ptrblck @xwang233 @ngimel