CUDA not available

username: MoRoBe
cooked: <p>I am aware variations of this have been asked multiple times, but even after working through many of those, I’m still stuck. I’m trying to get pytorch with CUDA support running on my Laptop. However, torch.cuda.is_available() returns False. Selected system information and diagnostic outputs are as follows:</p>
<p>Lenovo ThinkPad P14S Gen4<br>
NVIDIA RTX A500 Laptop GPU<br>
Linux Kernel 6.11.11-1<br>
NVIDIA Driver Version: 550.135</p>
<p>nvidia-smi output:</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.135                Driver Version: 550.135        CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX A500 Laptop GPU     Off |   00000000:03:00.0 Off |                  N/A |
| N/A   42C    P0              7W /   30W |       8MiB /   4096MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
</code></pre>
<p>nvcc --version:</p>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Tue_Oct_29_23:50:19_PDT_2024
Cuda compilation tools, release 12.6, V12.6.85
Build cuda_12.6.r12.6/compiler.35059454_0
</code></pre>
<p>torch.utils.collect_env:</p>
<pre><code class="lang-auto">PyTorch version: 2.5.1
Is debug build: False
CUDA used to build PyTorch: 12.6
ROCM used to build PyTorch: N/A

OS: Manjaro Linux (x86_64)
GCC version: (GCC) 14.2.1 20240910
Clang version: 18.1.8
CMake version: version 3.31.2
Libc version: glibc-2.40

Python version: 3.12.7 (main, Oct  1 2024, 11:15:50) [GCC 14.2.1 20240910] (64-bit runtime)
Python platform: Linux-6.11.11-1-MANJARO-x86_64-with-glibc2.40
Is CUDA available: False
CUDA runtime version: 12.6.85
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: GPU 0: NVIDIA RTX A500 Laptop GPU
Nvidia driver version: 550.135
cuDNN version: Probably one of the following:
/usr/lib/libcudnn.so.9.5.1
/usr/lib/libcudnn_adv.so.9.5.1
/usr/lib/libcudnn_cnn.so.9.5.1
/usr/lib/libcudnn_engines_precompiled.so.9.5.1
/usr/lib/libcudnn_engines_runtime_compiled.so.9.5.1
/usr/lib/libcudnn_graph.so.9.5.1
/usr/lib/libcudnn_heuristic.so.9.5.1
/usr/lib/libcudnn_ops.so.9.5.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
</code></pre>
<p>I’ve also tried using a both a venv and a conda env running pytorch 2.5.1 compiled against CUDA 12.4 with basically the same result.<br>
Not that it should make any difference, but CUDA is both in my PATH and my LD_LIBRARY_PATH.</p>
<p>As far as I understand the whole setup, versions should be matching and I really don’t understand what’s going wrong. Please let me know if you need any additional information!</p>
<p>Edit: I’ve also posted this question to StackOverflow (<a href="https://stackoverflow.com/questions/79358379/cuda-not-available" class="inline-onebox" rel="noopener nofollow ugc">pytorch - CUDA not available - Stack Overflow</a>) and will comment any solutions found there.</p>

username: MoRoBe
cooked: <p>I just found out torch.cuda.is_available() returns True in a docker container based on pytorch/pytorch2.1.2-cuda12.1-cudnn8-devel</p>
<p>For the moment I can simply work within this container, but would still prefer to also have CUDA available in my host systems pytorch.</p>

username: ptrblck
cooked: <p>It’s unclear what the issue is as you did not post any error.</p>
<p>The answer from your cross-post is also wrong:</p>
<blockquote>
<p>Maybe not the reason for the overall issue, but you should update your driver. Currently it only supports up to CUDA 12.4, so CUDA 12.6 is incompatible.</p>
</blockquote>
<p>since you don’t need to update the NVIDIA driver for newer minor CUDA versions due to <a href="https://docs.nvidia.com/deploy/cuda-compatibility/#minor-version-compatibility">Minor Version Compatibility</a>.</p>
<aside class="quote no-group" data-username="MoRoBe" data-post="1" data-topic="215422">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/morobe/48/39577_2.png" class="avatar"> MoRoBe:</div>
<blockquote>
<p>Not that it should make any difference, but CUDA is both in my PATH and my LD_LIBRARY_PATH.</p>
</blockquote>
</aside>
<p>Your locally installed CUDA toolkit won’t be used, since the PyTorch binaries ship with their own CUDA runtime dependencies.</p>

username: MoRoBe
cooked: <p>Hi, thanks for the reply, I was hoping for you, who appears to solve all of these problems, to show up. My follow-up is embarrassing even more so: out of a sudden, torch.cuda.is_available() returns True now. And, now that it does so, it’s working on the host, in both the venv and the conda env and also in another conda env using CUDA 12.1. Unfortunately for anyone ending up here via a search, I haven’t got the slightest idea what has changed in the meantime. So, sorry everyone for bothering you and thanks for the reply anyways!</p>

username: KFrank
cooked: <p>Hi Mo!</p>
<aside class="quote no-group quote-modified" data-username="MoRoBe" data-post="4" data-topic="215422" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/morobe/48/39577_2.png" class="avatar"> MoRoBe:</div>
<blockquote>
<p>out of a sudden, torch.cuda.is_available() returns True now.<br>
…<br>
I haven’t got the slightest idea what has changed in the meantime.</p>
</blockquote>
</aside>
<p>I have a laptop with a gpu similar to yours – ThinkPad P16v Gen 2, with<br>
the NVIDIA RTX 3000 Ada Generation Laptop GPU running Ubuntu (not<br>
Manjaro) LTS with kernel 22.04.1-49.</p>
<p>Note that the gpu driver – at least as used by cuda / pytorch – has some<br>
issues with ubuntu’s (and maybe linux’s, in general) power management,<br>
seemingly not restarting properly after a suspend.  You might try running<br>
pytorch with cuda immediately after rebooting or try <a class="mention" href="/u/ptrblck">@ptrblck</a>’s recipe<br>
for bouncing cuda:</p>
<blockquote>
<p>sudo rmmod nvidia_uvm<br>
sudo modprobe nvidia_uvm</p>
</blockquote>
<p>Best.</p>
<p>K. Frank</p>

username: MoRoBe
cooked: <p>I’ll give that a try tomorrow morning and report back! If only Linux, in the year 2025, had mastered suspend hooks… Thanks for the suggestion anyways, if that’s the problem I’d never have figured it out by myself!</p>

username: MoRoBe
cooked: <p>So directly after waking up from suspend this morning CUDA is still available. My next best guess would be that somethings going on regarding the docking station. For now I’ll mark <a class="mention" href="/u/kfrank">@KFrank</a> s answer as the best guess for a solution available.</p>

username: KFrank
cooked: <p>Hi Mo!</p>
<aside class="quote no-group" data-username="MoRoBe" data-post="7" data-topic="215422" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/morobe/48/39577_2.png" class="avatar"> MoRoBe:</div>
<blockquote>
<p>So directly after waking up from suspend this morning CUDA is still available.</p>
</blockquote>
</aside>
<p>Here is my best recipe for reproducing the bug.</p>
<p>First, test for cuda by creating (and maybe using) a tensor on the gpu.<br>
Sometimes <code>torch.cuda.is_available() = True</code> for me even when<br>
cuda has quit working:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; t = torch.randn (5, device = 'cuda')
&gt;&gt;&gt; t
tensor([ 2.0058, -0.0592,  1.4067, -1.8612, -0.4359], device='cuda:0')
&lt;*** after a lid-close suspend ***&gt;
&gt;&gt;&gt; torch.cuda.is_available()
True
&gt;&gt;&gt; t
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "&lt;path_to_pytorch_install&gt;/site-packages/torch/_tensor.py", line 523, in __repr__
    return torch._tensor_str._str(self, tensor_contents=tensor_contents)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "&lt;path_to_pytorch_install&gt;/site-packages/torch/_tensor_str.py", line 708, in _str
    return _str_intern(self, tensor_contents=tensor_contents)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "&lt;path_to_pytorch_install&gt;/torch/_tensor_str.py", line 625, in _str_intern
    tensor_str = _tensor_str(self, indent)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "&lt;path_to_pytorch_install&gt;/torch/_tensor_str.py", line 357, in _tensor_str
    formatter = _Formatter(get_summarized_data(self) if summarize else self)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "&lt;path_to_pytorch_install&gt;/torch/_tensor_str.py", line 145, in __init__
    nonzero_finite_vals = torch.masked_select(
                          ^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: unspecified launch failure
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
</code></pre>
<p>Recipe to reproduce:</p>
<p>Run python / pytorch and <strong>use cuda</strong>, for example, by creating a cuda<br>
tensor.  (Check that your python process running pytorch / cuda shows<br>
up in nvidia-smi.)  The suspend driver-bug doesn’t seem to occur unless<br>
cuda is already in use.</p>
<p>Close the laptop lid to trigger a suspend.</p>
<p>Come out of suspend.</p>
<p>Access / perform a computation with the cuda tensor.  At this point, i get<br>
a cuda error.  Note, <code>torch.cuda.is_available()</code> doesn’t necessary<br>
return <code>False</code> for me even though  cuda isn’t working.</p>
<p>Further color:</p>
<p>Killing and restarting the python process (so that you no longer see the<br>
python process in nvidia-smi) doesn’t fix cuda for me.  I can fix cuda<br>
(without a reboot) by killing the python process (and sometimes also<br>
nvidia-smi) to free the nvidia_uvm module, and then running:</p>
<blockquote>
<p>sudo rmmod nvidia_uvm<br>
sudo modprobe nvidia_uvm</p>
</blockquote>
<p>Note, I am under the impression (I haven’t tested this carefully.) that<br>
the gpu is sill working in the sense that I could run various graphics<br>
(e.g., gaming).  It’s just that cuda is broken.</p>
<p>Good luck!</p>
<p>K. Frank</p>