Cudnn_status_mapping_error

username: JuanFMontesinos
cooked: <p>Hi,<br>
I’ve got</p>
<pre><code class="lang-auto">  File "/home/jfm/.local/lib/python3.6/site-packages/flerken/framework/framework.py", line 206, in backpropagate
    self.loss.backward()
  File "/home/jfm/.local/lib/python3.6/site-packages/torch/tensor.py", line 185, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph)
  File "/home/jfm/.local/lib/python3.6/site-packages/torch/autograd/__init__.py", line 127, in backward
    allow_unreachable=True)  # allow_unreachable flag
cuDNN error: CUDNN_STATUS_MAPPING_ERROR
Exception raised from operator() at /pytorch/aten/src/ATen/native/cudnn/Conv.cpp:980 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f7e2106d1e2 in /home/jfm/.local/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: &lt;unknown function&gt; + 0xebae82 (0x7f7e22390e82 in /home/jfm/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #2: &lt;unknown function&gt; + 0xebcdb5 (0x7f7e22392db5 in /home/jfm/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #3: &lt;unknown function&gt; + 0xeb800e (0x7f7e2238e00e in /home/jfm/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #4: &lt;unknown function&gt; + 0xeb9bfb (0x7f7e2238fbfb in /home/jfm/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #5: at::native::cudnn_convolution_backward_input(c10::ArrayRef&lt;long&gt;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool) + 0xb2 (0x7f7e22390152 in /home/jfm/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #6: &lt;unknown function&gt; + 0xf1f35b (0x7f7e223f535b in /home/jfm/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #7: &lt;unknown function&gt; + 0xf4f178 (0x7f7e22425178 in /home/jfm/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #8: at::cudnn_convolution_backward_input(c10::ArrayRef&lt;long&gt;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool) + 0x1ad (0x7f7e5d2cd88d in /home/jfm/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #9: at::native::cudnn_convolution_backward(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, std::array&lt;bool, 2ul&gt;) + 0x223 (0x7f7e2238e823 in /home/jfm/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #10: &lt;unknown function&gt; + 0xf1f445 (0x7f7e223f5445 in /home/jfm/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #11: &lt;unknown function&gt; + 0xf4f1d4 (0x7f7e224251d4 in /home/jfm/.local/lib/python3.6/site-packages/torch/lib/libtorch_cuda.so)
frame #12: at::cudnn_convolution_backward(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, std::array&lt;bool, 2ul&gt;) + 0x1e2 (0x7f7e5d2dc242 in /home/jfm/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #13: &lt;unknown function&gt; + 0x2ec9c62 (0x7f7e5ef9fc62 in /home/jfm/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #14: &lt;unknown function&gt; + 0x2ede224 (0x7f7e5efb4224 in /home/jfm/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #15: at::cudnn_convolution_backward(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, c10::ArrayRef&lt;long&gt;, long, bool, bool, std::array&lt;bool, 2ul&gt;) + 0x1e2 (0x7f7e5d2dc242 in /home/jfm/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #16: torch::autograd::generated::CudnnConvolutionBackward::apply(std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt;&amp;&amp;) + 0x258 (0x7f7e5ee26c38 in /home/jfm/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #17: &lt;unknown function&gt; + 0x3375bb7 (0x7f7e5f44bbb7 in /home/jfm/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #18: torch::autograd::Engine::evaluate_function(std::shared_ptr&lt;torch::autograd::GraphTask&gt;&amp;, torch::autograd::Node*, torch::autograd::InputBuffer&amp;, std::shared_ptr&lt;torch::autograd::ReadyQueue&gt; const&amp;) + 0x1400 (0x7f7e5f447400 in /home/jfm/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #19: torch::autograd::Engine::thread_main(std::shared_ptr&lt;torch::autograd::GraphTask&gt; const&amp;) + 0x451 (0x7f7e5f447fa1 in /home/jfm/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #20: torch::autograd::Engine::thread_init(int,std::shared_ptr&lt;torch::autograd::ReadyQueue&gt; const&amp;, bool) + 0x89 (0x7f7e5f440119 in /home/jfm/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)
frame #21: torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr&lt;torch::autograd::ReadyQueue&gt; const&amp;, bool) + 0x4a (0x7f7e6cbe04ba in /home/jfm/.local/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #22: &lt;unknown function&gt; + 0xbd6df (0x7f7e6dd3c6df in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)
frame #23: &lt;unknown function&gt; + 0x76db (0x7f7e701f56db in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #24: clone + 0x3f (0x7f7e7052ea3f in /lib/x86_64-linux-gnu/libc.so.6)
</code></pre>
<p>Just wanted to know what may cause this error.</p>

username: user_123454321
cooked: <p>Does it work with lower batch sizes ? Also, can you run without cuda and check, I have found running on cpu produces more informative errors.</p>

username: JuanFMontesinos
cooked: <p>It’s a bit strange, the code runs for 2 epochs and then this error raises. That’s why I would like to understand what can cause it.</p>

username: ptrblck
cooked: <p>Could you post an executable code snippet to reproduce this issue as well as your setup (used GPU, CUDA, cudnn, PyTorch versions), so that we could have a look at it?</p>

username: JuanFMontesinos
cooked: <p>Hi,<br>
I’ve tried to reproduce it but I can’t.<br>
I just know that disabling cudnn fix it.<br>
It’s a very complex pipeline and I cannot identify the origin.</p>
<p>BTW is there anything useful in the error which gives me a hint about where to look for?</p>
<p>Soo it’s bit strange. Code runs for 2 epochs and then (no matter how large the epochs are) it raises this error. I’m using DALI library and I think it’s somehow related to instanting DALI’s pipeline epoch-wise.<br>
But i cannot really find what’s causing this.<br>
I find strange it occurs in the backward.</p>
<p>Soo if the error gives any clue it would be really useful.</p>

username: ptrblck
cooked: <p>You could enable <a href="https://docs.nvidia.com/deeplearning/cudnn/developer-guide/index.html#api-logging">cudnn API logging</a>, rerun the script, and post the output here, so that I could debug the issue and see, if cudnn is creating the crash or if maybe another sticky error is just caught by cudnn.<br>
Also, if possible, rerun the script via <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code>, which should point to the line of code, which causes the error.</p>

username: JuanFMontesinos
cooked: <p>You are gonna kill me but seems that<br>
<code>CUDA_LAUNCH_BLOCKING=1</code><br>
is not compatible with multi gpu.<br>
Aaand i reach a deadlock by using<br>
<code>CUDNN_LOGDEST_DBG=filename.txt</code><br>
The issue dissapears if I run everything in a single gpu<br>
</p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/9163" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/9163" target="_blank" rel="nofollow noopener">DataParallel model stucks with CUDA_LAUNCH_BLOCKING=1 sometime</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2018-07-04" data-time="07:54:22" data-timezone="UTC">07:54AM - 04 Jul 18 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2018-07-04" data-time="13:09:46" data-timezone="UTC">01:09PM - 04 Jul 18 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/acgtyrant" target="_blank" rel="nofollow noopener">
          <img alt="acgtyrant" src="https://avatars1.githubusercontent.com/u/3921062?v=4" class="onebox-avatar-inline" width="20" height="20">
          acgtyrant
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">Issue description
This is the context of the segment.py:
#!/usr/bin/env python
import torch
from torch import nn
import torch.utils.data

class DRN(nn.Module):
 def __init__(self):
 super(DRN, self).__init__()
 self.a =...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Soo thank you anyway for your help.</p>

username: ptrblck
cooked: <p>Sorry I didn’t realize you are using DDP, so skip the <code>CUDA_LAUNCH_BLOCKING=1</code> part (it might work, but could also create issues).<br>
Are you sure you are running into a dealock or did a GPU die and the other processes were waiting for the dead process?<br>
If you are using DDP, you could use <code>ps -auxf</code> and check the tree structure for a dead DDP process, when the script is hanging.<br>
If you see that a specific GPU-related process is indeed dead or hanging, you could send a <code>SIGHUP</code> to the main process, so that the logging might continue.</p>
<p>I understand that it’s not easy to create a reproducible code snippet, but if it would be possible, I could take care of the debugging, as it’s not always trivial.</p>