THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp

chrissikek: <p>Hey Guys,</p>
<p>I recently changed my gpu to an RTX2080ti and in the process also upgraded from CUDA8 to CUDA9 and the newest pytorch version. Ever since I get the following message when I use pytorch (the program keeps running though):</p>
<p>THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument</p>
<p>I could not find much on this and like to understand in what context this error can appear and if it can cause performance problems.</p>
<p>Thanks your help!</p>

albanD: <p>Hi,</p>
<p>Do the CUDA 9.0 samples work on an RTX2080 card? I though only CUDA 10 worked on these?</p>

chrissikek: <p>Thanks! You may be right, I have the exact same problem with the CUDA samples as described <a href="https://devtalk.nvidia.com/default/topic/1042638/is-nvidia-driver-410-57-incompatible-with-cuda-9-0/?offset=6#5289184" rel="nofollow noopener">here</a> , that indicates that there might be a fundamental problem with RTX2080ti and cuda9 (on linux?).</p>
<p>Would be interested if anyone has experienced otherwise.</p>

albanD: <p>I thinks it’s just that most recent cards like RTX2080Ti only work with CUDA 10 (not OS dependant).</p>

cristinasegalin: <p>I have the same problem with cuda 10 and pytorch 1.0</p>

ggeor: <p>Same problem for me. RTX2080Ti, Ubuntu 18.04, CUDA 10</p>

filippobistaffa: <p>I experience the same problem, I’m using CUDA 9 though.<br>
Anyway, it seems that the GPU is used anyway, even though I’m not sure if it’s used “fully”.</p>

ggeor: <p>By the way, for anyone following this topic, a solution to at least get things running is to set `torch.backends.cudnn.benchmark =  False’. I found this solution in this thread: <a href="https://discuss.pytorch.org/t/a-error-when-using-gpu/32761/20" class="inline-onebox">A error when using GPU</a></p>

chrissikek: <p>That may solve the problem of the error appearing, though it would be interesting to know why the rtx 2080 ti fails the cuda benchmark.</p>
<p>For me it does also fail the cudnn installation examples. Can anyone else confirm that?</p>

Felix_Lessange: <p>Same with cuda 10, ubuntu 18.04, and Titan RTX, on python 3.6</p>

Rifle_Zhang: <p>I put ‘torch.backends.cudnn.benchmark = False’<br>
at the beginning of my source code, but the error message still appears. Not sure why that happened.</p>

ggeor: <p>are you sure that later in the code that attribute is not set to true? try to print to screen torch.backends.cudnn.benchmark at various points of execution, to make sure it is indeed False</p>

Yiru_Shen: <p>Same issue here. Pytorch 1.0.0, CUDA 10.0, RTX 2080, on Fedora 28.<br>
Set " torch.backends.cudnn.benchmark=False" doesn’t work. It shows:<br>
cuDNN error: CUDNN_STATUS_EXECUTION_FAILED</p>
<p>Any information would be appreciated!</p>

Yiru_Shen: <p>Okay I have solve the problem. You cannot directly install pytorch, instead “pip3 install -U <a href="https://download.pytorch.org/whl/cu100/torch-1.0.0-cp36-cp36m-linux_x86_64.whl" rel="nofollow noopener">https://download.pytorch.org/whl/cu100/torch-1.0.0-cp36-cp36m-linux_x86_64.whl</a>” work for me.</p>

Evergrow: <p>Thanks your reply! But I want to know how to solve this problem on Pytorch 1.0.0, CUDA 9.0, RTX 2080. Must change to CUDA 10.0?</p>

ptrblck: <p>While some users seem to got the RTX 2080 working with CUDA9.X, it seems CUDA10 is the way to go for these new GPUs.</p>

zfzhang: <p>Hi, I signed up today to share a solution <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>My setup is 2080Ti, CUDA 10.1, python 3.6, installation method is pip3.</p>
<p>I originally installed the latest stable version suggested by pytorch home page  <a href="https://download.pytorch.org/whl/cu100/torch-1.0.1.post2-cp36-cp36m-linux_x86_64.whl" rel="nofollow noopener">https://download.pytorch.org/whl/cu100/torch-1.0.1.post2-cp36-cp36m-linux_x86_64.whl</a><br>
, which produces that error message.</p>
<p>Then I tried the version 1.0.0 suggested by <a href="https://discuss.pytorch.org/u/Yiru_Shen">Yiru_Shen</a>, which still produces that error message.</p>
<p>Finally I tried the nightly build, which does NOT produce any error message, luckily i don’t have to build from source <img src="https://discuss.pytorch.org/images/emoji/apple/grin.png?v=9" title=":grin:" class="emoji" alt=":grin:"></p>

ptrblck: <p>Welcome and thanks for the sign up and the information! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

zfzhang: <p>Thank you! This forum has really helped me a lot, and pytorch is the best framework I’ve seen so far in terms of code &amp; documentation <img src="https://discuss.pytorch.org/images/emoji/apple/smiley.png?v=9" title=":smiley:" class="emoji" alt=":smiley:">.</p>

jmaronas: <p>RTX only works with cuda 10 toolkit. I use this pytorch conda enviroment and it works (both python3.5 and python 3.7):</p>
<pre><code class="lang-auto">conda=/usr/local/anaconda3/bin/conda
activate=/usr/local/anaconda3/bin/activate
deactivate=/usr/local/anaconda3/bin/deactivate

#python3.7
$conda create -y --no-default-packages --prefix /usr/local/pytorch/python3.7/cuda10.0_pytorch_1.0.0 python=3.7
source $activate  /usr/local/pytorch/python3.7/cuda10.0_pytorch_1.0.0
$conda install -y pytorch=1.0.0 torchvision cuda100 -c pytorch
source $deactivate

#python 3.5
$conda create -y --no-default-packages --prefix /usr/local/pytorch/python3.5/cuda10.0_pytorch_1.0.0 python=3.5
source $activate  /usr/local/pytorch/python3.5/cuda10.0_pytorch_1.0.0
$conda install -y pytorch=1.0.0 torchvision cuda100 -c pytorch
source $deactivate


</code></pre>