Cuda error in Semantic Segmentation with FCN

Ankan_Sharma: <p>RuntimeError                              Traceback (most recent call last)<br>
 in ()<br>
15         labels=labels.long()<br>
16         lossi = loss(output, labels)<br>
—&gt; 17         lossi.backward()<br>
18         optimizer.step()<br>
19         print(f"batch {c}, loss–&gt;{lossi.item()}")</p>
<p>1 frames<br>
/usr/local/lib/python3.6/dist-packages/torch/autograd/<strong>init</strong>.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)<br>
125     Variable._execution_engine.run_backward(<br>
126         tensors, grad_tensors, retain_graph, create_graph,<br>
–&gt; 127         allow_unreachable=True)  # allow_unreachable flag<br>
128<br>
129</p>
<p>RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED<br>
Exception raised from createCuDNNHandle at /pytorch/aten/src/ATen/cudnn/Handle.cpp:9 (most recent call first):<br>
frame <span class="hashtag">#0:</span> c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f5b063391e2 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag">#1:</span>  + 0x100ca68 (0x7f5b077aea68 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#2:</span> at::native::getCudnnHandle() + 0x108d (0x7f5b077b034d in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#3:</span>  + 0xeda4cc (0x7f5b0767c4cc in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#4:</span>  + 0xed59ee (0x7f5b076779ee in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#5:</span>  + 0xed75db (0x7f5b076795db in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#6:</span> at::native::cudnn_convolution_backward_input(c10::ArrayRef, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef, c10::ArrayRef, c10::ArrayRef, long, bool, bool) + 0xb2 (0x7f5b07679b32 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#7:</span>  + 0xf3cd3b (0x7f5b076ded3b in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#8:</span>  + 0xf6cb58 (0x7f5b0770eb58 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#9:</span> at::cudnn_convolution_backward_input(c10::ArrayRef, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef, c10::ArrayRef, c10::ArrayRef, long, bool, bool) + 0x1ad (0x7f5b3e58588d in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#10:</span> at::native::cudnn_convolution_backward(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef, c10::ArrayRef, c10::ArrayRef, long, bool, bool, std::array&lt;bool, 2ul&gt;) + 0x223 (0x7f5b07678203 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#11:</span>  + 0xf3ce25 (0x7f5b076dee25 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#12:</span>  + 0xf6cbb4 (0x7f5b0770ebb4 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#13:</span> at::cudnn_convolution_backward(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef, c10::ArrayRef, c10::ArrayRef, long, bool, bool, std::array&lt;bool, 2ul&gt;) + 0x1e2 (0x7f5b3e594242 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#14:</span>  + 0x2ec9c62 (0x7f5b40257c62 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#15:</span>  + 0x2ede224 (0x7f5b4026c224 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#16:</span> at::cudnn_convolution_backward(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef, c10::ArrayRef, c10::ArrayRef, long, bool, bool, std::array&lt;bool, 2ul&gt;) + 0x1e2 (0x7f5b3e594242 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#17:</span> torch::autograd::generated::CudnnConvolutionBackward::apply(std::vector&lt;at::Tensor, std::allocator<a>at::Tensor</a> &gt;&amp;&amp;) + 0x258 (0x7f5b400dec38 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#18:</span>  + 0x3375bb7 (0x7f5b40703bb7 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#19:</span> torch::autograd::Engine::evaluate_function(std::shared_ptr<a>torch::autograd::GraphTask</a>&amp;, torch::autograd::Node*, torch::autograd::InputBuffer&amp;, std::shared_ptr<a>torch::autograd::ReadyQueue</a> const&amp;) + 0x1400 (0x7f5b406ff400 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#20:</span> torch::autograd::Engine::thread_main(std::shared_ptr<a>torch::autograd::GraphTask</a> const&amp;) + 0x451 (0x7f5b406fffa1 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#21:</span> torch::autograd::Engine::thread_init(int, std::shared_ptr<a>torch::autograd::ReadyQueue</a> const&amp;, bool) + 0x89 (0x7f5b406f8119 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#22:</span> torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr<a>torch::autograd::ReadyQueue</a> const&amp;, bool) + 0x4a (0x7f5b4de9834a in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag">#23:</span>  + 0xbd6df (0x7f5b6af1a6df in /usr/lib/x86_64-linux-gnu/libstdc++.so.6)<br>
frame <span class="hashtag">#24:</span>  + 0x76db (0x7f5b6bffc6db in /lib/x86_64-linux-gnu/libpthread.so.0)<br>
frame <span class="hashtag">#25:</span> clone + 0x3f (0x7f5b6c335a3f in /lib/x86_64-linux-gnu/libc.so.6)</p>
<p>These above is the error<br>
output = fcn_resnet_50(images)[“out”]</p>
<pre><code>    labels=labels.long()

    lossi = loss(output, labels)

    lossi.backward()

    optimizer.step()
</code></pre>
<p>number of classes match with number of levels<br>
link of colab<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://colab.research.google.com/img/favicon.ico?vrz=colab-20201008-085601-RC00_336084530" class="site-icon" width="16" height="16">
      <a href="https://colab.research.google.com/drive/1lx_xRlRXUhQe0Osoo5TGrEaLUJyf-voK?usp=sharing" target="_blank" rel="noopener nofollow ugc">colab.research.google.com</a>
  </header>
  <article class="onebox-body">
    <img src="https://colab.research.google.com/img/colab_favicon_256px.png" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://colab.research.google.com/drive/1lx_xRlRXUhQe0Osoo5TGrEaLUJyf-voK?usp=sharing" target="_blank" rel="noopener nofollow ugc">Google Colaboratory</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


ptrblck: <p>Could you post an executable code snippet to reproduce this issue, please?<br>
Also, which PyTorch and CUDA versions are you using?</p>

Ankan_Sharma: <p>Thank you for the quick reply. But it worked out, I was giving number of classes wrong. So that is the reason it is giving CUDA error</p>