Error: CUDNN_STATUS_EXECUTION_FAILED

ms15: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> . ANy help on this is highly appreciated. The problem is when I switchecd to RTX 5060Ti.</p>
<p>System Config:<br>
OS : 22.04 Ubuntu<br>
Python Version: Python-3.11.13<br>
Pytorch: torch-2.9.0.dev20250727+cu129<br>
CUDA:12.9 (NVIDIA GeForce RTX 5060 Ti, 15848MiB)<br>
cuDNN: 9.11.0( as in /home/robo/robo_env/lib/python3.11/site-packages/nvidia/cudnn/include/cudnn_version.h)</p>
<p>I’m getting the below error</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/robo/face_detect/main.py", line 464, in &lt;module&gt;
    main(opt)
  File "/home/robo/face_detect/main.py", line 457, in main
    run(**vars(opt))
  File "/home/robo/robo_env/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/robo/face_detect/main.py", line 218, in run
    model.warmup(imgsz=(1 if pt or model.triton else bs, 3, *imgsz))
  File "/home/robo/face_detect/models/common.py", line 771, in warmup
    self.forward(im)  # warmup
    ^^^^^^^^^^^^^^^^
  File "/home/robo/face_detect/models/common.py", line 688, in forward
    y = self.model(im, augment=augment, visualize=visualize) if augment or visualize else self.model(im)
                                                                                          ^^^^^^^^^^^^^^
  File "/home/robo/robo_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/robo/robo_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/robo/face_detect/models/yolo.py", line 270, in forward
    return self._forward_once(x, profile, visualize)  # single-scale inference, train
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/robo/face_detect/models/yolo.py", line 169, in _forward_once
    x = m(x)  # run
        ^^^^
  File "/home/robo/robo_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/robo/robo_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/robo/face_detect/models/common.py", line 247, in forward
    return self.cv3(torch.cat((self.m(self.cv1(x)), self.cv2(x)), 1))
                               ^^^^^^^^^^^^^^^^^^^
  File "/home/robo/robo_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/robo/robo_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/robo/robo_env/lib/python3.11/site-packages/torch/nn/modules/container.py", line 250, in forward
    input = module(input)
            ^^^^^^^^^^^^^
  File "/home/robo/robo_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/robo/robo_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/robo/face_detect/models/common.py", line 181, in forward
    return x + self.cv2(self.cv1(x)) if self.add else self.cv2(self.cv1(x))
                                                      ^^^^^^^^^^^^^^^^^^^^^
  File "/home/robo/robo_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/robo/robo_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/robo/face_detect/models/common.py", line 87, in forward
    return self.act(self.bn(self.conv(x)))
                            ^^^^^^^^^^^^
  File "/home/robo/robo_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/robo/robo_env/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/robo/robo_env/lib/python3.11/site-packages/torch/nn/modules/conv.py", line 548, in forward
    return self._conv_forward(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/robo/robo_env/lib/python3.11/site-packages/torch/nn/modules/conv.py", line 543, in _conv_forward
    return F.conv2d(
           ^^^^^^^^^
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
</code></pre>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the issue (ideally using random data), please?</p>

ms15: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a><br>
Thanks for your response. Normal pytorch operations like <code>matmul , conv2d runs with out any issue. I have checked it using nvidia-smi. However when I use libraries like hugging face</code> <code>operations take long time and I can see using htop</code> that everything is getting executed using CPU nothing in the GPU. The above error is when I run inference on YOLOv5. WHy such behaviour? Is this a problem from pytorch end? Now RTX 40 series is not available in the market. That is a problem. I don’t understand what is the problem. Any help is highly appreciated.</p>