Running MASK-RCNN 3D demo in my conda env, a error occurs-"RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

username: ShiChenchen
cooked: <p>when running MASK-RCNN 3D demo in my conda env, a error occurs-“RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED<br>
Backend Qt5Agg is interactive backend. Turning interactive mode on.<br>
Opening ZED Camera…<br>
Mask enabled!<br>
Traceback (most recent call last):<br>
File “C:\Users\Lenovo.conda\envs\pytorch_test\lib\runpy.py”, line 197, in _run_module_as_main<br>
return _run_code(code, main_globals, None,<br>
File “C:\Users\Lenovo.conda\envs\pytorch_test\lib\runpy.py”, line 87, in <em>run_code<br>
exec(code, run_globals)<br>
File "c:\Users\Lenovo.vscode\extensions\ms-python.python-2021.10.1365161279\pythonFiles\lib\python\debugpy_<em>main</em></em>.py”, line 45, in <br>
cli.main()<br>
File “c:\Users\Lenovo.vscode\extensions\ms-python.python-2021.10.1365161279\pythonFiles\lib\python\debugpy/…\debugpy\server\cli.py”, line 444, in main<br>
run()<br>
File “c:\Users\Lenovo.vscode\extensions\ms-python.python-2021.10.1365161279\pythonFiles\lib\python\debugpy/…\debugpy\server\cli.py”, line 285, in run_file<br>
runpy.run_path(target_as_str, run_name=compat.force_str(“<strong>main</strong>”))<br>
File “C:\Users\Lenovo.conda\envs\pytorch_test\lib\runpy.py”, line 268, in run_path<br>
return _run_module_code(code, init_globals, run_name,<br>
File “C:\Users\Lenovo.conda\envs\pytorch_test\lib\runpy.py”, line 97, in _run_module_code<br>
_run_code(code, mod_globals, init_globals,<br>
File “C:\Users\Lenovo.conda\envs\pytorch_test\lib\runpy.py”, line 87, in _run_code<br>
exec(code, run_globals)<br>
File “d:\CloudDisk\OneDrive - <a href="http://nuaa.edu.cn" rel="noopener nofollow ugc">nuaa.edu.cn</a>\15_code\03_Python_Code\02_ZEDCamera\zed-pytorch-master\zed-pytorch-master\zed_object_detection.py”, line 290, in <br>
main()<br>
File “d:\CloudDisk\OneDrive - <a href="http://nuaa.edu.cn" rel="noopener nofollow ugc">nuaa.edu.cn</a>\15_code\03_Python_Code\02_ZEDCamera\zed-pytorch-master\zed-pytorch-master\zed_object_detection.py”, line 254, in main<br>
prediction = coco_demo.select_top_predictions(coco_demo.compute_prediction(img))<br>
File “d:\CloudDisk\OneDrive - <a href="http://nuaa.edu.cn" rel="noopener nofollow ugc">nuaa.edu.cn</a>\15_code\03_Python_Code\02_ZEDCamera\zed-pytorch-master\zed-pytorch-master\predictor.py”, line 246, in compute_prediction<br>
predictions = self.model(image_list)<br>
File “C:\Users\Lenovo.conda\envs\pytorch_test\lib\site-packages\torch\nn\modules\module.py”, line 1102, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “C:\Users\Lenovo.conda\envs\pytorch_test\lib\site-packages\maskrcnn_benchmark-0.1-py3.9-win-amd64.egg\maskrcnn_benchmark\modeling\detector\generalized_rcnn.py”, line 50, in forward<br>
proposals, proposal_losses = self.rpn(images, features, targets)<br>
File “C:\Users\Lenovo.conda\envs\pytorch_test\lib\site-packages\torch\nn\modules\module.py”, line 1102, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “C:\Users\Lenovo.conda\envs\pytorch_test\lib\site-packages\maskrcnn_benchmark-0.1-py3.9-win-amd64.egg\maskrcnn_benchmark\modeling\rpn\rpn.py”, line 155, in forward<br>
objectness, rpn_box_regression = self.head(features)<br>
File “C:\Users\Lenovo.conda\envs\pytorch_test\lib\site-packages\torch\nn\modules\module.py”, line 1102, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “C:\Users\Lenovo.conda\envs\pytorch_test\lib\site-packages\maskrcnn_benchmark-0.1-py3.9-win-amd64.egg\maskrcnn_benchmark\modeling\rpn\rpn.py”, line 103, in forward<br>
t = F.relu(self.conv(feature))<br>
File “C:\Users\Lenovo.conda\envs\pytorch_test\lib\site-packages\torch\nn\modules\module.py”, line 1102, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “C:\Users\Lenovo.conda\envs\pytorch_test\lib\site-packages\torch\nn\modules\conv.py”, line 446, in forward<br>
return self._conv_forward(input, self.weight, self.bias)<br>
File “C:\Users\Lenovo.conda\envs\pytorch_test\lib\site-packages\torch\nn\modules\conv.py”, line 442, in _conv_forward<br>
return F.conv2d(input, weight, bias, self.stride,<br>
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED<br>
You can try to repro this exception using the following code snippet. If that doesn’t trigger the error, please include your original repro script when reporting this issue.</p>
<p>import torch<br>
torch.backends.cuda.matmul.allow_tf32 = True<br>
torch.backends.cudnn.benchmark = False<br>
torch.backends.cudnn.deterministic = False<br>
torch.backends.cudnn.allow_tf32 = True<br>
data = torch.randn([1, 256, 16, 30], dtype=torch.float, device=‘cuda’, requires_grad=True)<br>
net = torch.nn.Conv2d(256, 256, kernel_size=[3, 3], padding=[1, 1], stride=[1, 1], dilation=[1, 1], groups=1)<br>
net = net.cuda().float()<br>
out = net(data)<br>
out.backward(torch.randn_like(out))<br>
torch.cuda.synchronize()</p>
<p>ConvolutionParams<br>
data_type = CUDNN_DATA_FLOAT<br>
padding = [1, 1, 0]<br>
stride = [1, 1, 0]<br>
dilation = [1, 1, 0]<br>
groups = 1<br>
deterministic = false<br>
allow_tf32 = true<br>
input: TensorDescriptor 0000029E84679850<br>
type = CUDNN_DATA_FLOAT<br>
nbDims = 4<br>
dimA = 1, 256, 16, 30,<br>
strideA = 122880, 480, 30, 1,<br>
output: TensorDescriptor 0000029E8467AC70<br>
type = CUDNN_DATA_FLOAT<br>
nbDims = 4<br>
dimA = 1, 256, 16, 30,<br>
strideA = 122880, 480, 30, 1,<br>
weight: FilterDescriptor 0000029E9D71DF70<br>
type = CUDNN_DATA_FLOAT<br>
tensor_format = CUDNN_TENSOR_NCHW<br>
nbDims = 4<br>
dimA = 256, 256, 3, 3,<br>
Pointer addresses:<br>
input: 0000000BA9278000<br>
output: 0000000BA92F0000<br>
weight: 0000000B1C5C0000<br>
Forward algorithm: 1</p>
<p>I tried to solve the problem many times, but unfortunately failed.</p>
<p>1、Disable the CUDNN module in “predictor.py”<br>
– torch.backends.cudnn.enabled=False</p>
<p>2、Check the edition of host cuda and conda env cuda, and reinstall cudatoolkit 11.3.1, cuda and conda installration info list as below:</p>
<p>My System info:</p>
<blockquote>
<blockquote>
<blockquote>
<p>OS-WIN10;<br>
CUDA VERSION-11.3.109<br>
GPU-RTX3080-16GB;</p>
</blockquote>
</blockquote>
</blockquote>
<p>My Develop info:</p>
<blockquote>
<blockquote>
<blockquote>
<p>conda env name - pytorch_test;<br>
conda package list<br>
cudatoolkit-11.3.1</p>
</blockquote>
</blockquote>
</blockquote>
<p>could you plz give me some advice!!!</p>

username: ptrblck
cooked: <p>What kind of error did you see when disabling cuDNN? If another error was raised, this could mean that cuDNN would also be running into this issue, although it’s not necessarily the case.</p>

username: ShiChenchen
cooked: <p>Thank you, after disable the cudnn in main function, the program works fine.<br>
At first I only disabled the cudnn in module section.</p>

username: ptrblck
cooked: <p>Could you post an executable code snippet to reproduce the issue, please?<br>
I thought the issue might be related to <a href="https://discuss.pytorch.org/t/transform-failed-to-synchronize-cudaerrorillegaladdress-an-illegal-memory-access-was-encountered/136028/8">this one</a>, which I’m already debugging, but since disabling cuDNN seems to work I doubt it.</p>