CUDA 11.4 and torch version 1.11.0 not working

Mans: <p>Hi there,</p>
<p>I have CUDA 11.4 and I’ve installed pytorch 1.11.0.<br>
I thought these versions where compatible but I get this error when I run a python script : <code>RuntimeError: CUDA error: no kernel image is available for execution on the device</code>.</p>
<p>When I run t<code>orch.cuda.is_available()</code> I get <code>True</code>.</p>
<p>Is the problem coming from the CUDA and pytorch versions I’ve installed ?</p>
<p>I have a NVIDIA RTX A4000.</p>

ptrblck: <p>I guess you’ve installed the pip wheels or conda binaries, which will ship with their own CUDA runtime.<br>
Your locally installed CUDA toolkit will be used if you build PyTorch from source or custom CUDA extensions.<br>
Based on the error message you’ve most likely installed the binary with the CUDA 10.2 runtime while your Ampere GPU needs CUDA &gt;= 11.</p>

Mans: <p>Thank you for your answer.<br>
The CUDA Runtime Version that I’ve installed is 11.3.</p>
<p>If it can help, here’s what I get when I display the deviceQuery :</p>
<pre><code class="lang-auto">./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "NVIDIA RTX A4000"
  CUDA Driver Version / Runtime Version          11.4 / 11.3
  CUDA Capability Major/Minor version number:    8.6
  Total amount of global memory:                 16095 MBytes (16876699648 bytes)
  (48) Multiprocessors, (128) CUDA Cores/MP:     6144 CUDA Cores
  GPU Max Clock rate:                            1560 MHz (1.56 GHz)
  Memory Clock rate:                             7001 Mhz
  Memory Bus Width:                              256-bit
  L2 Cache Size:                                 4194304 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  1536
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device supports Compute Preemption:            Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 1 / 0
  Compute Mode:
     &lt; Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) &gt;

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 11.4, CUDA Runtime Version = 11.3, NumDevs = 1, Device0 = NVIDIA RTX A4000
Result = PASS
</code></pre>
<p>I really don’t see where the problem is…</p>

ptrblck: <p>As described before, the binaries will use their own Runtime, which is given by:</p>
<pre><code class="lang-python">print(torch.version.cuda)
</code></pre>
<p>and I guess it would return <code>10.2</code>, which is incompatible with your GPU as it’s missing the desired compute capability.</p>
<p>In this case, select the CUDA 11 version for the pip wheel / conda binaries and reinstall it.</p>

Mans: <p>Hello,</p>
<p><code>torch.version.cuda</code> returns : <code>11.3</code> .<br>
So it’s supposed to be compatible right ?</p>

ptrblck: <p>Yes, 11.3 should work and I also just verified that the binaries with the 11.3 runtime work properly on an A4000 using:</p>
<pre><code class="lang-python">import torch

print(torch.__version__)
print(torch.version.cuda)
print(torch.backends.cudnn.version())
print(torch.cuda.get_device_name(0))
print(torch.cuda.get_device_properties(0))

x = torch.randn(1, 3, 224, 224, device='cuda')
conv = torch.nn.Conv2d(3, 3, 3).cuda()

out = conv(x)
print(out.shape)
</code></pre>
<p>Output:</p>
<pre><code class="lang-python">1.11.0+cu113
11.3
8200
NVIDIA RTX A4000
_CudaDeviceProperties(name='NVIDIA RTX A4000', major=8, minor=6, total_memory=16117MB, multi_processor_count=48)
torch.Size([1, 3, 222, 222])
</code></pre>

Mans: <p>I get the exact same results except for <code>print(torch.__version__)</code><br>
where I get only <code>1.11.0</code> as an output.<br>
Is it a problem ?</p>

ptrblck: <p>Is the code snippet working as well or are you seeing the same error message when executing it?</p>

Mans: <p>The code is working properly yes… Do you know if pytorch is available with CUDA 11.6 ?<br>
I’ve tried to change CUDA version to 11.6 and I’ve installed pytorch with the command</p>
<blockquote>
<p>conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch</p>
</blockquote>
<p>(the latest version), but it doesn’t seem to work either. So I’m wondering if it needs CUDA 11.3.</p>

ptrblck: <aside class="quote no-group" data-username="Mans" data-post="9" data-topic="152416">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/439d5e/48.png" class="avatar"> Mans:</div>
<blockquote>
<p>The code is working properly yes…</p>
</blockquote>
</aside>
<p>In that case I don’t fully understand what is causing the issue with the missing GPU architecture.<br>
Are you using a custom extension and didn’t build it for Ampere GPUs?</p>
<aside class="quote no-group" data-username="Mans" data-post="9" data-topic="152416">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/439d5e/48.png" class="avatar"> Mans:</div>
<blockquote>
<p>Do you know if pytorch is available with CUDA 11.6 ?</p>
</blockquote>
</aside>
<p>Yes, we have internal CUDA 11.6 wheels, which can be downloaded e.g. via:</p>
<pre data-code-wrap="python"><code class="lang-python">pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116
</code></pre>
<p>but this would not fix the issue you are seeing as you are either using the CUDA 10.2 wheels (installed via a wrongly specified requirement) or are building an extension for older architectures.</p>

Mans: <aside class="quote no-group" data-username="ptrblck" data-post="10" data-topic="152416">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Are you using a custom extension and didn’t build it for Ampere GPUs?</p>
</blockquote>
</aside>
<p>Sorry, I don’t really know what a custom extension is.<br>
My script is written in python and runs a flask application. It is a python’s framework used to create web applications easily.<br>
What I’m trying to do is to display some images on this web app. Those images are passed through a neural network.<br>
This code work well on a colleague’s computer who have a different GPU and a different version of CUDA.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="10" data-topic="152416">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>but this would not fix the issue you are seeing as you are either using the CUDA 10.2 wheels (installed via a wrongly specified requirement) or are building an extension for older architectures.</p>
</blockquote>
</aside>
<p>I’ve installed CUDA 11.6 following these installation instructions :</p>
<pre><code class="lang-auto">wget https://developer.download.nvidia.com/compute/cuda/11.6.0/local_installers/cuda_11.6.0_510.39.01_linux.run
sudo sh cuda_11.6.0_510.39.01_linux.run

</code></pre>

Mans: <p>Hello again !</p>
<blockquote>
<p>UserWarning:<br>
NVIDIA RTX A4000 with CUDA capability sm_86 is not compatible with the current PyTorch installation.<br>
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.</p>
</blockquote>
<p>This was in my console too. Is it what you meant with GPU architectures ?</p>

Mans: <p>So I just found the solution to my issue. I had installed the wrong pytorch version for my gpu architecture (I think).<br>
I installed pytorch 1.8.1 which is compatible with the architecture sm_86 and now it’s working fine !<br>
Thank you <a class="mention" href="/u/ptrblck">@ptrblck</a> for helping me !!</p>

spyroot: <p>I don’t think <a href="https://download.pytorch.org/whl/cu116/" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu116/</a> available.  If you will do get request it access denied while <a href="https://download.pytorch.org/whl/cu115" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu115</a> show a listing of all packages.</p>

DOW_digital_ops: <p>Ok everyone after reading and rereading this thread trying to figure out what in the heck I did wrong… This is the real answer!</p>
<p>Pip is installing torch from the cache of downloads that you have previously installed.</p>
<ol start="0">
<li>
<p>UNINSTALL torch in Terminal: pip3 uninstall torch</p>
</li>
<li>
<p>DELETE the pip folder in ~/.cache</p>
</li>
<li>
<p>Rerun installation of torch and cuda from this link <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a></p>
</li>
</ol>
<p>You’re welcome in advance!!</p>

Zakir: <p>Whatever version I Install in my environment it shows successfully installed but when I check with print(torch.<strong>version</strong>) it shows me 1.12.1+cu113 every time. can anyone please help me?</p>

ptrblck: <p>You might have multiple PyTorch version installed and <code>1.12.1+cu113</code> might be the “default” one your current environment finds. Uninstall all previous installations via <code>pip uninstall torch -y</code> and <code>conda uninstall pytorch</code> and rerun these commands until no PyTorch install can be found, then reinstall the desired one.</p>