A error when using GPU

zhangying1230: <p>The error is<br>
“THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=405 error=11 : invalid argument”. But it doesn’t influence the training and test, I want to know the reason for this error.<br>
My cuda version is 9.0 and the python version is  3.6.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/5/5221e87848dd128871696b52cb91c97b38cfef78.png" data-download-href="https://discuss.pytorch.org/uploads/default/5221e87848dd128871696b52cb91c97b38cfef78" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/2X/5/5221e87848dd128871696b52cb91c97b38cfef78.png" alt="image" data-base62-sha1="bIzNCppurxjhOAjNaQgWAWU41OU" width="690" height="229" data-dominant-color="101111"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">698×232 5.59 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Thank you for help <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=12" title=":smile:" class="emoji" alt=":smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <p>Does your code just run with this error at the beginning?<br>
If so, are you using multiple GPUs?<br>
Could you try to run your code with <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code>?</p>
<p>I’ve never seen a “silent” CUDA error. Usually my code just explodes with error messages when I mess up some CUDA calls. <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=6" title=":smiley:" class="emoji" alt=":smiley:"></p>

yang_lee: <p>hello, does your preblem is solved? I have also met the preblem</p>

zhangying1230: <p>I have tried the code “CUDA_LAUNCH_BLOCKING=1 python all_resnet.py” , but it doesn’t work. : (<br>
Thank you very much for reply! Merry Christmas:smile:</p>

zhangying1230: <p>No,  I haven’t solved it yet : (</p>
<p>Merry Christmas!</p>

ptrblck: <p>Does your code crash with some error message or does it just get stuck, if you use <code>CUDA_LAUNCH_BLOCKING=1</code>?</p>
<p>Merry Christmas! <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"></p>

Huang_Wade: <p>Hi,all<br>
I met the same problem. My cuda version is 10.0 and the python version is 3.6.<br>
I got error message is “RuntimeError: cuda runtime error (11) : invalid argument at /pytorch/aten/src/THC/THCGeneral.cpp:405”</p>
<p>After I used CUDA_LAUNCH_BLOCKING=1 python script.py args, it only used one GPU to work.</p>
<p>Dose anyone have solution?<br>
Merry Christmas!</p>

ptrblck: <p>Could you post the complete stack trace with the lines of code this error is pointing to?<br>
Also, is your code working on the CPU?</p>

andys0975: <p>It seems like some conflicts occur when using pytorch 1.0 with RTX2080(Ti) series.<br>
I used the same code and the same docker image to run, and this kind of error only happened on RTX2080Ti server. (My docker image is built up from cuda10-cudnn7.4 image and the driver version is 410.79)</p>

andys0975: <p>I found the source of this problem: cudnn.benchmark</p>

ptrblck: <p>Is it working, if you disable <code>cudnn.banchmark</code> or what do you mean?</p>

andys0975: <p>I guess that using cudnn.banchmark=True with pytorch 1.0 + RTX2080 + Dockerfile cuda10.0-cudnn7-devel-ubuntu18.04 would cause this error.</p>

zhangying1230: <p>It has the same performance as without using <code>CUDA_LAUNCH_BLOCKING=1</code>,  the problem still exists, but no other error.<br>
Thank you so much for reply !</p>

ptrblck: <p>The <code>CUDA_LAUNCH_BLOCKING=1</code> env variable just makes sure to call all CUDA operations synchronously, so that an error message should point to the right line of code in the stack trace.<br>
Did you get any errors? If so, could you post the stack trace?</p>

zhangying1230: <p>The error message is “THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=405 error=11 : invalid argument”. But I can’t find the THCGeneral.cpp.</p>

ptrblck: <p>Is your code running fine on the CPU? Could you post the whole stack trace?</p>

heilaw: <p>I am also having the same issue with 2080Ti.</p>
<p>If I enable <code>cudnn.benchmark</code>, my code gives the error and crashes. If I disable <code>cudnn.benchmark</code>, my code still gives the same error but it can still run. Adding <code>CUDA_LAUNCH_BLOCKING=1</code> doesn’t give anymore details. It stills shows that the code crashes when it reaches the first convolution.</p>
<p>The code was running fine on 1080Ti with <code>cudnn.benchmark</code> enabled.</p>

jclevesque: <p>Same here, I get this error message on my RTX 2080 Ti but not on the 1080 Ti, same Pytorch (1.0.0) and CUDA (10.0.130), python 3.5.2.</p>
<p>Code to produce the warning/error:</p>
<pre><code class="lang-auto">import os
import torch

# force torch to use my RTX 2080TI GPU, modify or remove accordingly
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"] = "1"
torch.backends.cudnn.benchmark = True

from torchvision.models import vgg16
model = vgg16().cuda()
x = torch.zeros((32, 3, 227, 227)).cuda()
model(x)
</code></pre>
<p>Prints the error (<code>THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=405 error=11 : invalid argument</code>) and doesn’t return anything.</p>

jwickens: <p>I have a RTX 2070 with CUDA 10, pytorch 1.0, python 3.6 on Ubuntu 18 and I get this error when running this project: <a href="https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix" rel="nofollow noopener">https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix</a></p>
<p>With  <code>torch.backends.cudnn.benchmark = True</code> I get the below stack trace and the program exits.</p>
<pre><code class="lang-auto">THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument
Traceback (most recent call last):
  File "test.py", line 60, in &lt;module&gt;
    model.test()           # run inference
  File "/home/jwickens/dev/face-translation/pytorch-CycleGAN-and-pix2pix/models/base_model.py", line 105, in test
    self.forward()
  File "/home/jwickens/dev/face-translation/pytorch-CycleGAN-and-pix2pix/models/test_model.py", line 65, in forward
    self.fake_B = self.netG(self.real_A)  # G(A)
  File "/home/jwickens/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jwickens/.local/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 121, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/home/jwickens/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jwickens/dev/face-translation/pytorch-CycleGAN-and-pix2pix/models/networks.py", line 399,in forward
    return self.model(input)
  File "/home/jwickens/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jwickens/.local/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward
    input = module(input)
  File "/home/jwickens/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/jwickens/.local/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 301, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: cuda runtime error (11) : invalid argument at /pytorch/aten/src/THC/THCGeneral.cpp:663
</code></pre>
<p>Without that line I get a silent CUDA error once at the beginning. The script works though. <code>THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument</code></p>
<p>I also have the same silent error with this tutorial <a href="https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a></p>
<p>There are quite a few issues out here for this error message, some users say its cuda 9.2 and others RTX cards.</p>

ggeor: <p>RTX 2080ti with cuda 10.0. I got the same problem. I followed the advice of others and turned `torch.backends.cudnn.benchmark = True’ to False and things started working again</p>