RuntimeError cuDNN error: CUDNN_STATUS_BAD_PARAM

CR7: <p>When I use two GPUs for training I got the following error, :<br>
Exception has occurred: RuntimeError<br>
cuDNN error: CUDNN_STATUS_BAD_PARAM (operator() at /opt/conda/conda-bld/pytorch_1587428398394/work/aten/src/ATen/native/cudnn/Conv.cpp:1142)</p>
<p>My codes are as follows:</p>
<pre><code>use_gpu = torch.cuda.is_available()
if use_gpu and torch.cuda.device_count() &gt; 1:
   net = nn.DataParallel(net)
   net.cuda()
   net.train()

for i, (images, target) in enumerate(train_loader):
    if use_gpu:
       images, target = images.cuda(), target.cuda()
    pred = net(images)
    loss = criterion(pred, target)
    total_loss += loss.data.item()
    optimizer.zero_grad()
    loss.backward()
</code></pre>
<p>Error when running loss.backward()</p>
<p>Exception has occurred: RuntimeError<br>
cuDNN error: CUDNN_STATUS_BAD_PARAM (operator() at /opt/conda/conda-bld/pytorch_1587428398394/work/aten/src/ATen/native/cudnn/Conv.cpp:1142)<br>
frame <span class="hashtag">#0:</span> c10::Error::Error(c10::SourceLocation, std::string const&amp;) + 0x4e (0x7f3c98237b5e in /home/cr7/anaconda3/lib/python3.7/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag">#1:</span>  + 0xd792a2 (0x7f3c991dd2a2 in /home/cr7/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#2:</span>  + 0xd76225 (0x7f3c991da225 in /home/cr7/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#3:</span>  + 0xd776bf (0x7f3c991db6bf in /home/cr7/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#4:</span>  + 0xd7b310 (0x7f3c991df310 in /home/cr7/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#5:</span> at::native::cudnn_convolution_backward_weight(c10::ArrayRef, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef, c10::ArrayRef, c10::ArrayRef, long, bool, bool) + 0x49 (0x7f3c991df569 in /home/cr7/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#6:</span>  + 0xde1ec0 (0x7f3c99245ec0 in /home/cr7/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#7:</span>  + 0xe26138 (0x7f3c9928a138 in /home/cr7/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#8:</span> at::native::cudnn_convolution_backward(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, c10::ArrayRef, c10::ArrayRef, c10::ArrayRef, long, bool, bool, std::array&lt;bool, 2ul&gt;) + 0x2fc (0x7f3c991e021c in /home/cr7/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#9:</span>  + 0xde1bcb (0x7f3c99245bcb in /home/cr7/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#10:</span>  + 0xe26194 (0x7f3c9928a194 in /home/cr7/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#11:</span>  + 0x29defc6 (0x7f3cc1df7fc6 in /home/cr7/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#12:</span>  + 0x2a2ea54 (0x7f3cc1e47a54 in /home/cr7/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#13:</span> torch::autograd::generated::CudnnConvolutionBackward::apply(std::vector&lt;at::Tensor, std::allocator<a>at::Tensor</a> &gt;&amp;&amp;) + 0x378 (0x7f3cc1a0ff28 in /home/cr7/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#14:</span>  + 0x2ae8215 (0x7f3cc1f01215 in /home/cr7/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#15:</span> torch::autograd::Engine::evaluate_function(std::shared_ptr&lt;<a>Torch::autograd::GraphTask</a>&gt;&amp;, torch::autograd::Node*, torch::autograd::InputBuffer&amp;) + 0x16f3 (0x7f3cc1efe513 in /home/cr7/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#16:</span> torch::autograd::Engine::thread_main(std::shared_ptrTorch::autograd::GraphTasK&gt;&gt; const&amp;, bool) + 0x3d2 (0x7f3cc1eff2f2 in /home/cr7/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#17:</span> torch::autograd::Engine::thread_init(int) + 0x39 (0x7f3cc1ef7969 in /home/cr7/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#18:</span> torch::autograd::python::PythonEngine::thread_init(int) + 0x38 (0x7f3cc523e558 in /home/cr7/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag">#19:</span>  + 0xc819d (0x7f3cfad4d19d in /home/cr7/anaconda3/bin/…/lib/libstdc++.so.6)<br>
frame <span class="hashtag">#20:</span>  + 0x76db (0x7f3d042ae6db in /lib/x86_64-linux-gnu/libpthread.so.0)<br>
frame <span class="hashtag">#21:</span> clone + 0x3f (0x7f3d03fd788f in /lib/x86_64-linux-gnu/libc.so.6)<br>
File “/home/cr7/AI/github_projects/pytorch-YOLO-v1-master/train.py”, line 136, in <br>
loss.backward()</p>
<p>By the way, I tried to train the network on CPU only, it works just fine.</p>
<p>My software environment and hardware type are as follows:<br>
python: 3.7<br>
pytorch: 1.5.0<br>
cuda: 10.1.168<br>
cudnn: 7.6.2.24</p>
<p>The two GPUs I use are GeForce RTX 2070 and GeForce GTX 1060</p>

albanD: <p>Hi,</p>
<p>Can you share a small code sample that we can use to reproduce this?<br>
Also what are the exact parameters of the convolution that causes the error (input size, kernel size, stride, padding, etc)?<br>
Thanks !</p>

CR7: <p>Thank you for your reply!<br>
This is the repository I used:<br>
</p><aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/xiongzihua/pytorch-YOLO-v1" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars1.githubusercontent.com/u/28775927?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/xiongzihua/pytorch-YOLO-v1" target="_blank" rel="nofollow noopener">xiongzihua/pytorch-YOLO-v1</a></h3>

<p>an experiment for yolo-v1, including training and testing. - xiongzihua/pytorch-YOLO-v1</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
But I tried to call nn.DataParallel() to implement multiple GPUs training follow the instructions below:<br>
<a href="https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html#imports-and-parameters" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/blitz/data_parallel_tutorial.html#imports-and-parameters</a><br>
And then I got the error mentioned above, I’m not sure whether it is because the type  of two GPUs  I use is different, but I’m doubting…

CR7: <p>Hi, I found this problem sometimes does not appear, but most of the time it will report an error. If I gradually debug and run the program in IDE(eg, vscode), it can be backpropagated normally. If I run the program in the terminal or directly in IDE, this error will basically be thrown.</p>

ptrblck: <p>Could you update to PyTorch <code>1.5.1</code> and CUDA10.2.89 + cudnn7.6.5.32?<br>
Also, is the YOLO code working fine and the error is raised by your dataparallel code?<br>
If so, could you post the code to reproduce this error (with input shapes if possible)?</p>