RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED (batch_norm)

username: Arpit_Garg
cooked: <p>RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
15     pretrained = pretrained,<br>
16     dataset=dataset,<br>
—&gt; 17     noise_rate=0.45<br>
18 )</p>
<p> in run_vae(train_loader, test_loader, batch_size, epochs, z_dim, est_loader, cls_model, out_dir, select_ratio, pretrained, dataset, noise_rate)<br>
57<br>
58         adjust_learning_rate(optimizers[‘vae2’], epoch)<br>
—&gt; 59         train(epoch, model, train_loader, optimizers, device)<br>
60<br>
61</p>
<p> in train(epoch, model, train_loader, optimizers, device)<br>
19<br>
20         <span class="hashtag">#forward</span><br>
—&gt; 21         x_hat1, n_logits1, mu1, log_var1, c_logits1, y_hat1  = vae_model1(data)<br>
22         x_hat2, n_logits2, mu2, log_var2, c_logits2, y_hat2 = vae_model2(data)<br>
23         <span class="hashtag">#calculate</span> acc</p>
<p>/home/ubuntu/anaconda3/envs/idnl/lib/python3.6/site-packages/torch/nn/modules/module.py in <strong>call</strong>(self, *input, **kwargs)<br>
545             result = self._slow_forward(*input, **kwargs)<br>
546         else:<br>
 → 547             result = self.forward(*input, **kwargs)<br>
548         for hook in self._forward_hooks.values():<br>
549             hook_result = hook(self, input, result)</p>
<p>/media/ubuntu/Storage/Noisy_Labels/IDLN/mylib/models/vae.py in forward(self, x)<br>
32     def forward(self, x):<br>
33         ### trick 1, add a softmax function to logits<br>
—&gt; 34         c_logits = self.y_encoder(x)<br>
35         y_hat = self._y_hat_reparameterize(c_logits)<br>
36         mu, logvar = self.z_encoder(x, y_hat)</p>
<p>/home/ubuntu/anaconda3/envs/idnl/lib/python3.6/site-packages/torch/nn/modules/module.py in <strong>call</strong>(self, *input, **kwargs)<br>
545             result = self._slow_forward(*input, **kwargs)<br>
546         else:<br>
 → 547             result = self.forward(*input, **kwargs)<br>
548         for hook in self._forward_hooks.values():<br>
549             hook_result = hook(self, input, result)</p>
<p>/media/ubuntu/Storage/Noisy_Labels/IDLN/mylib/models/resnet.py in forward(self, x, revision, output_f)<br>
220<br>
221     def forward(self, x, revision=False, output_f=False):<br>
 → 222         return self._forward_impl(x,revision,output_f)<br>
223<br>
224</p>
<p>/media/ubuntu/Storage/Noisy_Labels/IDLN/mylib/models/resnet.py in _forward_impl(self, x, revision, output_f)<br>
199         # See note [TorchScript super()]<br>
200         x = self.conv1(x)<br>
 → 201         x = self.bn1(x)<br>
202         x = self.relu(x)<br>
203         x = self.maxpool(x)</p>
<p>/home/ubuntu/anaconda3/envs/idnl/lib/python3.6/site-packages/torch/nn/modules/module.py in <strong>call</strong>(self, *input, **kwargs)<br>
545             result = self._slow_forward(*input, **kwargs)<br>
546         else:<br>
 → 547             result = self.forward(*input, **kwargs)<br>
548         for hook in self._forward_hooks.values():<br>
549             hook_result = hook(self, input, result)</p>
<p>/home/ubuntu/anaconda3/envs/idnl/lib/python3.6/site-packages/torch/nn/modules/batchnorm.py in forward(self, input)<br>
79             input, self.running_mean, self.running_var, self.weight, self.bias,<br>
80             self.training or not self.track_running_stats,<br>
—&gt; 81             exponential_average_factor, self.eps)<br>
82<br>
83     def extra_repr(self):</p>
<p>/home/ubuntu/anaconda3/envs/idnl/lib/python3.6/site-packages/torch/nn/functional.py in batch_norm(input, running_mean, running_var, weight, bias, training, momentum, eps)<br>
1654     return torch.batch_norm(<br>
1655         input, weight, bias, running_mean, running_var,<br>
 → 1656         training, momentum, eps, torch.backends.cudnn.enabled<br>
1657     )<br>
1658</p>
<p>RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED</p>
<p>I am running this on pytorch==1.2 and I have tried both cuda 9.0 and 10.0; can’t figure out the error<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>

username: wmpauli
cooked: <p>Did you ever figure out what’s was causing the Runtime Error. I’m running into a similar issue when using DataParallel.</p>

username: ptrblck
cooked: <p>Would you post a minimal and executable code snippet to reproduce the issue using the latest stable or nightly release, please?</p>

username: wmpauli
cooked: <p>Thank you for your response (as always).</p>
<p>This only happens when I use data <code>model = nn.DataParallel(model)</code>, not on single device.</p>
<p>If I disable cudnn (<code>torch.backends.cudnn.enabled = False</code>), I don’t get an error either.</p>
<p>after setting env <code>"CUDA_LAUNCH_BLOCKING": "1"</code>, the error appears to occur in line (this is not my code).</p><aside class="onebox githubblob" data-onebox-src="https://github.com/zhouhao94/CANet/blob/a011a76009243d50082eb6b7c631eb2d16f61dc6/Co_attention.py#L137">
  <header class="source">

      <a href="https://github.com/zhouhao94/CANet/blob/a011a76009243d50082eb6b7c631eb2d16f61dc6/Co_attention.py#L137" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/zhouhao94/CANet/blob/a011a76009243d50082eb6b7c631eb2d16f61dc6/Co_attention.py#L137" target="_blank" rel="noopener nofollow ugc">zhouhao94/CANet/blob/a011a76009243d50082eb6b7c631eb2d16f61dc6/Co_attention.py#L137</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="127" style="counter-reset: li-counter 126 ;">
          <li>gap_p = sum([gap_1, y])
</li>
          <li>gap_c = sum([gap_2, z])
</li>
          <li>
</li>
          <li>gap_p = F.adaptive_avg_pool2d(gap_p, 1)  # n, 512, h, w -&gt; n, 512, 1, 1
</li>
          <li>gap_c = F.adaptive_avg_pool2d(gap_c, 1)  # n, 512, h, w -&gt; n, 512, 1, 1
</li>
          <li>
</li>
          <li>gap_p = self.fc1_p(gap_p)
</li>
          <li>gap_c = self.fc1_c(gap_c)
</li>
          <li>
</li>
          <li>if self.use_bn:
</li>
          <li class="selected">    gap_p = self.bn1_p(gap_p)
</li>
          <li>    gap_c = self.bn1_c(gap_c)
</li>
          <li>
</li>
          <li>gap_p = self.relu(gap_p)
</li>
          <li>gap_c = self.relu(gap_c)
</li>
          <li>
</li>
          <li>atten_p = self.fc2_p(gap_p)  # n, 256, 1, 1 -&gt; n, 1024, 1, 1
</li>
          <li>atten_c = self.fc2_c(gap_c)  # n, 256, 1, 1 -&gt; n, 1024, 1, 1
</li>
          <li>
</li>
          <li>atten_p = self.rsoftmax(atten_p).view(batch, -1, 1, 1)  # (n, 1024) -&gt; (n, 1024, 1, 1)
</li>
          <li>atten_c = self.rsoftmax(atten_c).view(batch, -1, 1, 1)  # (n, 1024) -&gt; (n, 1024, 1, 1)
</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>gap_p looks like this when it happens.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/6/66836d3a09636eb235ae2152c346ee5c4bbe9c0f.png" data-download-href="https://discuss.pytorch.org/uploads/default/66836d3a09636eb235ae2152c346ee5c4bbe9c0f" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/6/66836d3a09636eb235ae2152c346ee5c4bbe9c0f.png" alt="image" data-base62-sha1="eCShveXqSqifE0ncTz5QiDIBkar" width="505" height="500" data-dominant-color="222220"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">884×875 34.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><strong>Note:</strong> the error occurs the second time this line is executed, on device:1. No problem the first time.</p>
<p>Traceback:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "/home/foo_user/miniconda3/envs/also_foo/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/foo_user/miniconda3/envs/also_foo/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/foo_user/.vscode-server/extensions/ms-python.python-2022.16.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 39, in &lt;module&gt;
    cli.main()
  File "/home/foo_user/.vscode-server/extensions/ms-python.python-2022.16.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 430, in main
    run()
  File "/home/foo_user/.vscode-server/extensions/ms-python.python-2022.16.1/pythonFiles/lib/python/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 284, in run_file
    runpy.run_path(target, run_name="__main__")
  File "/home/foo_user/.vscode-server/extensions/ms-python.python-2022.16.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 321, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "/home/foo_user/.vscode-server/extensions/ms-python.python-2022.16.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 135, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "/home/foo_user/.vscode-server/extensions/ms-python.python-2022.16.1/pythonFiles/lib/python/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 124, in _run_code
    exec(code, run_globals)
  File "train.py", line 302, in &lt;module&gt;
    main()
  File "train.py", line 184, in main
    loss, acc = train(model, clf, dataloader, crit)
  File "train.py", line 43, in train
    out = model(input)
  File "/home/foo_user/miniconda3/envs/also_foo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/foo_user/miniconda3/envs/also_foo/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 168, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/home/foo_user/miniconda3/envs/also_foo/lib/python3.9/site-packages/torch/nn/parallel/data_parallel.py", line 178, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/home/foo_user/miniconda3/envs/also_foo/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py", line 86, in parallel_apply
    output.reraise()
  File "/home/foo_user/miniconda3/envs/also_foo/lib/python3.9/site-packages/torch/_utils.py", line 461, in reraise
    raise exception
RuntimeError: Caught RuntimeError in replica 1 on device 1.
Original Traceback (most recent call last):
  File "/home/foo_user/miniconda3/envs/also_foo/lib/python3.9/site-packages/torch/nn/parallel/parallel_apply.py", line 61, in _worker
    output = module(*input, **kwargs)
  File "/home/foo_user/miniconda3/envs/also_foo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/foo_user/foo/model.py", line 61, in forward
    fusion = self.co_attention(x, y)
  File "/home/foo_user/miniconda3/envs/also_foo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/foo_user/foo/model.py", line 102, in forward
    out = self.split_conv(m, p_out, c_out)
  File "/home/foo_user/miniconda3/envs/also_foo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/foo_user/foo/co_attention.py", line 137, in forward
    gap_p = self.bn1_p(gap_p)
  File "/home/foo_user/miniconda3/envs/also_foo/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/foo_user/miniconda3/envs/also_foo/lib/python3.9/site-packages/torch/nn/modules/batchnorm.py", line 168, in forward
    return F.batch_norm(
  File "/home/foo_user/miniconda3/envs/also_foo/lib/python3.9/site-packages/torch/nn/functional.py", line 2438, in batch_norm
    return torch.batch_norm(
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
</code></pre>
<p>Here is the result of <code>$ python -m torch.utils.collect_env </code></p>
<pre data-code-wrap="Collecting"><code class="lang-Collecting">PyTorch version: 1.12.1+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.15.5
Libc version: glibc-2.31

Python version: 3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:58:50)  [GCC 10.3.0] (64-bit runtime)
Python platform: Linux-5.15.0-1022-azure-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: 
GPU 0: Tesla V100-PCIE-16GB
GPU 1: Tesla V100-PCIE-16GB

Nvidia driver version: 470.103.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.2
[pip3] torch==1.12.1+cu113
[pip3] torchvision==0.13.1+cu113
[conda] cudatoolkit               11.3.1              h9edb442_10    conda-forge
[conda] numpy                     1.23.2                   pypi_0    pypi
[conda] torch                     1.12.1+cu113             pypi_0    pypi
[conda] torchvision               0.13.1+cu113             pypi_0    pypi
</code></pre>
<p>For now, I can just disable batch_normalization to avoid the problem. Model seems to converge well without.</p>

username: ptrblck
cooked: <p>Thanks for sharing the repository. Could you let me know how to execute it using random input tensors to reproduce the issue, please?<br>
Also, do you see the issue using the latest PyTorch release with CUDA 11.7?</p>

username: wmpauli
cooked: <p>I’ll try to get to reproducing with code snippet. For now I can say that the error also occurs with nightly build, with cu116 torch1140.</p>
<p>Getting an environment with cu117 will be difficult, because I depends on docker containers that somebody else builds. Thanks again.</p>