:RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

Mohan_Krishna: <p>[ Fri Feb 28 20:05:08 2025 ] Parameters:<br>
{‘work_dir’: ‘./work_dir/baseline_res18/’, ‘config’: ‘./configs/baseline.yaml’, ‘random_fix’: True, ‘device’: ‘0’, ‘phase’: ‘train’, ‘save_interval’: 5, ‘random_seed’: 0, ‘eval_interval’: 1, ‘print_log’: True, ‘log_interval’: 10000, ‘evaluate_tool’: ‘sclite’, ‘feeder’: ‘dataset.dataloader_video.BaseFeeder’, ‘dataset’: ‘phoenix2014-T’, ‘dataset_info’: {‘dataset_root’: ‘./dataset/phoenix2014-T’, ‘dict_path’: ‘./preprocess/phoenix2014-T/gloss_dict.npy’, ‘evaluation_dir’: ‘./evaluation/slr_eval’, ‘evaluation_prefix’: ‘phoenix2014-T-groundtruth’}, ‘num_worker’: 5, ‘feeder_args’: {‘mode’: ‘test’, ‘datatype’: ‘video’, ‘num_gloss’: -1, ‘drop_ratio’: 1.0, ‘frame_interval’: 1, ‘image_scale’: 1.0, ‘input_size’: 224, ‘prefix’: ‘./dataset/phoenix2014-T’, ‘transform_mode’: False}, ‘model’: ‘slr_network.SLRModel’, ‘model_args’: {‘num_classes’: 1116, ‘c2d_type’: ‘resnet18’, ‘conv_type’: 2, ‘use_bn’: 1, ‘share_classifier’: True, ‘weight_norm’: True}, ‘load_weights’: None, ‘load_checkpoints’: None, ‘decode_mode’: ‘beam’, ‘ignore_weights’: <span class="chcklst-box fa fa-square-o fa-fw"></span>, ‘batch_size’: 1, ‘test_batch_size’: 1, ‘loss_weights’: {‘SeqCTC’: 1.0, ‘ConvCTC’: 1.0, ‘Dist’: 25.0}, ‘optimizer_args’: {‘optimizer’: ‘Adam’, ‘base_lr’: 0.0001, ‘step’: [20, 35], ‘learning_ratio’: 1, ‘weight_decay’: 0.0001, ‘start_epoch’: 0, ‘nesterov’: False}, ‘num_epoch’: 40}</p>
<p>0%|                                                                                                                                                          | 0/7096 [00:01&lt;?, ?it/s]<br>
Traceback (most recent call last):<br>
File “/media/mohan/mohan/CorrNet/main.py”, line 255, in <br>
processor.start()<br>
File “/media/mohan/mohan/CorrNet/main.py”, line 67, in start<br>
seq_train(self.data_loader[‘train’], self.model, self.optimizer,<br>
File “/media/mohan/mohan/CorrNet/seq_scripts.py”, line 28, in seq_train<br>
loss = model.criterion_calculation(ret_dict, label, label_lgt)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/media/mohan/mohan/CorrNet/slr_network.py”, line 122, in criterion_calculation<br>
loss += weight * self.loss[‘CTCLoss’](ret_dict[“sequence_logits”].log_softmax(-1),<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/media/mohan/anaconda3/envs/corrnet/lib/python3.12/site-packages/torch/nn/modules/module.py”, line 1736, in _wrapped_call_impl<br>
return self._call_impl(*args, **kwargs)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/media/mohan/anaconda3/envs/corrnet/lib/python3.12/site-packages/torch/nn/modules/module.py”, line 1747, in _call_impl<br>
return forward_call(*args, **kwargs)<br>
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
File “/media/mohan/anaconda3/envs/corrnet/lib/python3.12/site-packages/torch/nn/modules/loss.py”, line 1980, in forward<br>
return F.ctc_loss(<br>
^^^^^^^^^^^<br>
File “/media/mohan/anaconda3/envs/corrnet/lib/python3.12/site-packages/torch/nn/functional.py”, line 3069, in ctc_loss<br>
return torch.ctc_loss(<br>
^^^^^^^^^^^^^^^<br>
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED</p>

ptrblck: <p>If your code still fails in the latest nightly or stable release, could you post a minimal and executable code snippet, please?</p>

Mohan_Krishna: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/a/7a2f4fba7c7cbc7457eefaaffc4fb650d67fd77f.png" data-download-href="https://discuss.pytorch.org/uploads/default/7a2f4fba7c7cbc7457eefaaffc4fb650d67fd77f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7a2f4fba7c7cbc7457eefaaffc4fb650d67fd77f_2_690x457.png" alt="image" data-base62-sha1="hqTCaHdjZU5shIWPm614lvvwA6H" width="690" height="457" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7a2f4fba7c7cbc7457eefaaffc4fb650d67fd77f_2_690x457.png, https://discuss.pytorch.org/uploads/default/original/3X/7/a/7a2f4fba7c7cbc7457eefaaffc4fb650d67fd77f.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/a/7a2f4fba7c7cbc7457eefaaffc4fb650d67fd77f.png 2x" data-dominant-color="3A172F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">921×610 80.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>(corrnet) mohan@debanga-Precision-7820-Tower:/media/mohan/mohan/CorrNet$ python<br>
Python 3.12.9 | packaged by Anaconda, Inc. | (main, Feb  6 2025, 18:56:27) [GCC 11.2.0] on linux<br>
Type “help”, “copyright”, “credits” or “license” for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch</p>
<p>print(“Torch version:”, torch.<strong>version</strong>)<br>
Torch version: 2.5.1+cu121<br>
print(“CUDA version:”, torch.version.cuda)<br>
CUDA version: 12.1<br>
print(“cuDNN version:”, torch.backends.cudnn.version())<br>
cuDNN version: 90701<br>
exit(0<br>
… )<br>
(corrnet) mohan@debanga-Precision-7820-Tower:/media/mohan/mohan/CorrNet$ nvcc --version<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2023 NVIDIA Corporation<br>
Built on Mon_Apr__3_17:16:06_PDT_2023<br>
Cuda compilation tools, release 12.1, V12.1.105<br>
Build cuda_12.1.r12.1/compiler.32688072_0</p>
</blockquote>
</blockquote>
</blockquote>
<p>I am currently working on this repo</p>
<p>the following is the code of slr_network.py</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/hulianyuyy/CorrNet/blob/main/slr_network.py">
  <header class="source">

      <a href="https://github.com/hulianyuyy/CorrNet/blob/main/slr_network.py" target="_blank" rel="noopener nofollow ugc">github.com/hulianyuyy/CorrNet</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/hulianyuyy/CorrNet/blob/main/slr_network.py" target="_blank" rel="noopener nofollow ugc">slr_network.py</a></h4>

<div class="git-blob-info">
  <a href="https://github.com/hulianyuyy/CorrNet/blob/main/slr_network.py" rel="noopener nofollow ugc"><code>main</code></a>
</div>


      <pre><code class="lang-py">import pdb
import copy
import utils
import torch
import types
import numpy as np
import torch.nn as nn
import torch.nn.functional as F
import torchvision.models as models
from modules.criterions import SeqKD
from modules import BiLSTMLayer, TemporalConv
import modules.resnet as resnet

class Identity(nn.Module):
    def __init__(self):
        super(Identity, self).__init__()

    def forward(self, x):
        return x

</code></pre>



  This file has been truncated. <a href="https://github.com/hulianyuyy/CorrNet/blob/main/slr_network.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>the following is my main.py file:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/hulianyuyy/CorrNet/blob/main/main.py">
  <header class="source">

      <a href="https://github.com/hulianyuyy/CorrNet/blob/main/main.py" target="_blank" rel="noopener nofollow ugc">github.com/hulianyuyy/CorrNet</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/hulianyuyy/CorrNet/blob/main/main.py" target="_blank" rel="noopener nofollow ugc">main.py</a></h4>

<div class="git-blob-info">
  <a href="https://github.com/hulianyuyy/CorrNet/blob/main/main.py" rel="noopener nofollow ugc"><code>main</code></a>
</div>


      <pre><code class="lang-py">import os

os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
import pdb
import sys
import cv2
import yaml
import torch
import random
import importlib
import faulthandler
import numpy as np
import torch.nn as nn
import shutil
import inspect
import time
from collections import OrderedDict

faulthandler.enable()
import utils
</code></pre>



  This file has been truncated. <a href="https://github.com/hulianyuyy/CorrNet/blob/main/main.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>