Help why torch.cuda.is_available return True but my GPU didn't work

yutianfanxing: <p>Cuda driver is the 11.2 and the cudatoolkit is 11.0 and after import torch  torch.cuda.is_available return True</p>

ptrblck: <p>How did you check that the GPU isn’t working?<br>
What does <code>nvidia-smi</code> show after you push the model and inputs to the GPU and execute a forward pass?</p>

progers: <p>Similar issues.</p>
<p>PyTorch version: 1.7.1+cu110<br>
Cuda version: NVIDIA-SMI 460.27.04    Driver Version: 460.27.04    CUDA Version: 11.2<br>
(This is the latest beta driver for Ubuntu; needed to fix persistent driver crash.)<br>
torch.cuda.is_available(): True<br>
When running a model: RuntimeError: CUDA error: no kernel image is available for execution on the device</p>
<p>Now, I am a PyTorch/Cuda newbie, so user error is a real possibility.</p>
<p>What are the compatibility rules for Cuda/PyTorch? Should the Cuda 11.0 PyTorch build work with 11.2? A doc on the Nvidia web site suggests it should, FWIW. Or, should I rebuild from sources as suggested for other instances of this issue?</p>

ptrblck: <p>Your local CUDA installation won’t be used, if you are installing the binaries, which ship with their own CUDA runtime. You would thus only need the appropriate driver.</p>
<p>The error message is raised, if you are using a build, which doesn’t support your GPU architecture, so which GPU are you using?</p>

yutianfanxing: <p>well ， i have used a RTX2070s  and after I install back my cuda version to 10.2 and pytorch version to 1.6 it just works so I think the newest version don’t have good compatibility</p>

ptrblck: <p>I cannot reproduce this error using the <code>1.7.1</code> binaries with CUDA11.0 as well as CUDA10.2 and this code snippet on an RTX2070:</p>
<pre><code class="lang-python">import torch

print(torch.cuda.get_device_name())
print(torch.__version__)
print(torch.version.cuda)
x = torch.randn(1).cuda()
print(x)
</code></pre>
<p>Result:</p>
<pre><code class="lang-python">GeForce RTX 2070
1.7.1
11.0
tensor([1.7284], device='cuda:0')
...
GeForce RTX 2070
1.7.1
10.2
tensor([0.8304], device='cuda:0')
</code></pre>
<p>so the binaries should contain the needed architecture code.</p>

progers: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, thanks much for your response. You asked about my GPU:</p>
<pre><code class="lang-auto">  In: torch.cuda.get_device_name()
  Out: GeForce GT 710
</code></pre>
<p>Found <a href="https://developer.nvidia.com/cuda-gpus#compute" rel="noopener nofollow ugc">this link</a> to supported Cuda products; the GT 710 is not listed. Yet, the product box claims Cuda support, nvidia-smi gives the info listed earlier and the Nvidia UI claims it has 192 Cuda cores.</p>
<p>Maybe I was a bit too cheap in getting the lowest-cost GPU that supports both a 4K screen and (supposedly) Cuda…</p>
<p>So, the first step is to verify that the GT 710 does support Cuda. Is there a non-PyTorch way to do that?</p>
<p>If that checks out, I can try upgrading to the just-released 460 driver. (Upgrading Nvidia drivers on Linux Mint is a nightmare.)</p>

ptrblck: <p>Your GT710 supports CUDA and uses a compute capability of 3.5.<br>
The PyTorch binaries are built for a compute capability &gt;= 3.7 as seen <a href="https://github.com/pytorch/builder/blob/aa5725692987a644123b875313b0ec9b66296dc7/conda/pytorch-nightly/build.sh#L32-L46">here</a>, so you would need to build PyTorch from source as described <a href="https://github.com/pytorch/pytorch#from-source">here</a>.</p>

progers: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, thanks for the quick answer. I’ll look into doing the compile as you suggest.</p>
<p>I wonder, does it make sense to add a “cudaAudit()” function that could do all these checks and print out problems? Or, is there one already?</p>

ptrblck: <p>You can check the built-in compute capabilities in the currently used binary via:</p>
<pre><code class="lang-python">print(torch.cuda.get_arch_list())
</code></pre>

progers: <p>My goal is to learn PyTorch, hence my attempt to use my existing low-end GPU card to get started. For any others who might find this: <a href="https://timdettmers.com/2020/09/07/which-gpu-for-deep-learning" rel="noopener nofollow ugc">a bit of reading</a> suggests that was a naive idea.</p>
<p>I’ll try “plan B”: use the CPU and <a href="https://pytorch.org/" rel="noopener nofollow ugc">offload to the cloud</a> for occasional heavy lifting until I’m ready to invest in a proper high-end card for serious work.</p>

ptrblck: <p>I don’t think learning PyTorch would be bottlenecked by your “low-end” GPU and you could certainly learn all basic concepts. Of course you won’t be able to train huge models on large datasets, but it also depends what exactly you try to learn first.</p>

theAayushbajaj: <p>facing the same issue</p>
<blockquote>
<p>GPU:1080Ti<br>
Cuda: 10.1 (nvcc -V)</p>
</blockquote>
<p>tried pytorch 1.6, 1.7.0+cu101, 1.7.1+cu101<br>
The same settings were working previously but after NVIDIA driver update to 460.39 it stopped working.</p>

ptrblck: <p>Do you see an error while running your script? If so, could you post the error message including the complete stack trace?</p>

theAayushbajaj: <p>No there’s no error, the program returns true for:</p>
<blockquote>
<p>torch.cuda.is_available()</p>
</blockquote>
<p>but runs the code on cpu. GPU usage remains ~0% on nvidia-smi</p>

ptrblck: <p>If you are transferring the data to the GPU via <code>model.cuda()</code> or <code>model.to('cuda')</code>, the GPU will be used. Otherwise an error would be raised.<br>
A low GPU utilization might come from different bottlenecks in your code, e.g. the data loading.<br>
As a quick check you could run a simple matrix multiplication on the GPU in a loop and should see some GPU utilization in <code>nvidia-smi</code>.</p>

theAayushbajaj: <p><code>model.to('cuda')</code> is being used for that. I’ll check with the matmul operation.</p>

theAayushbajaj: <p>I think I’ve found the issue, you’re right its different bottlenecks in the code. Thanks!</p>

Ceyhun_Efe_Kayan: <p>I am facing the same issue, torch.cuda.is_available() returns TRUE, torch.cuda.get_device_name(torch.cuda.current_device()) returns NVIDIA GeForce GTX 1060 but code is running on the CPU.</p>

ptrblck: <p>Did you push the data and model to the GPU? If so and if you haven’t received an error, the GPU will be used. In case the utilization is low, you could profile you code and check where the bottleneck might be (e.g. maybe in the data loading).</p>