Torch.is_available is False on Cuda 11.7

VictorJouault: <p>Common issue, but I really cannot root cause this one.</p>
<p>Trying to set up PyTorch + Cuda on an AWS p3 instance (Nvidia Tesla V100 GPUs).</p>
<p>Torch output:</p>
<pre><code class="lang-auto">torch.__version__               # 2.0.1+cu117
torch.cuda.device_count()       # --&gt; 0
torch.cuda.is_available()       # --&gt; False
torch.version.cuda              # --&gt; 11.7
torch.backends.cudnn.version()  # 8500
torch.zeros(1).cuda()           # “RuntimeError: Found no NVIDIA driver on your system”
</code></pre>
<p><code>pip list</code> output:</p>
<pre><code class="lang-auto">Package                  Version
------------------------ ----------
cmake                    3.26.3
filelock                 3.12.0
Jinja2                   3.1.2
lit                      16.0.5
MarkupSafe               2.1.2
mpmath                   1.3.0
networkx                 3.1
nvidia-cublas-cu11       11.10.3.66
nvidia-cuda-cupti-cu11   11.7.101
nvidia-cuda-nvrtc-cu11   11.7.99
nvidia-cuda-runtime-cu11 11.7.99
nvidia-cudnn-cu11        8.5.0.96
nvidia-cufft-cu11        10.9.0.58
nvidia-curand-cu11       10.2.10.91
nvidia-cusolver-cu11     11.4.0.1
nvidia-cusparse-cu11     11.7.4.91
nvidia-nccl-cu11         2.14.3
nvidia-nvtx-cu11         11.7.91
pip                      22.3.1
setuptools               65.6.3
sympy                    1.12
torch                    2.0.1
triton                   2.0.0
typing_extensions        4.6.2
wheel                    0.40.0
</code></pre>
<p>nvidia-smi output</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.105.01   Driver Version: 515.105.01   CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...  Off  | 00000000:00:17.0 Off |                    0 |
| N/A   33C    P0    57W / 300W |      0MiB / 16384MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...  Off  | 00000000:00:18.0 Off |                    0 |
| N/A   32C    P0    56W / 300W |      0MiB / 16384MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Tesla V100-SXM2...  Off  | 00000000:00:19.0 Off |                    0 |
| N/A   33C    P0    56W / 300W |      0MiB / 16384MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  Tesla V100-SXM2...  Off  | 00000000:00:1A.0 Off |                    0 |
| N/A   34C    P0    55W / 300W |      0MiB / 16384MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   4  Tesla V100-SXM2...  Off  | 00000000:00:1B.0 Off |                    0 |
| N/A   33C    P0    55W / 300W |      0MiB / 16384MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   5  Tesla V100-SXM2...  Off  | 00000000:00:1C.0 Off |                    0 |
| N/A   32C    P0    56W / 300W |      0MiB / 16384MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   6  Tesla V100-SXM2...  Off  | 00000000:00:1D.0 Off |                    0 |
| N/A   32C    P0    58W / 300W |      0MiB / 16384MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   7  Tesla V100-SXM2...  Off  | 00000000:00:1E.0 Off |                    0 |
| N/A   33C    P0    55W / 300W |      0MiB / 16384MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
</code></pre>
<p><code>nvcc --version</code> output</p>
<pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_May__3_18:49:52_PDT_2022
Cuda compilation tools, release 11.7, V11.7.64
Build cuda_11.7.r11.7/compiler.31294372_0
</code></pre>
<p>Also, I’ve downloaded the Nvidia samples using <code>git clone https://github.com/NVIDIA/cuda-samples.git --branch v11.6</code> and then ran them using <code>make</code>, and they all seem to run fine …</p>
<p>Thanks.</p>
<p><strong>Edit</strong>:</p>
<ol>
<li>ChatGPT suggested I check the permissions, so here is the output although I don’t find anything specific:</li>
</ol>
<p>Running <code>ls -l /dev/nvidia*</code></p>
<pre><code class="lang-auto">crw-rw-rw- 1 root root   195,   0 Jun  1 02:56 /dev/nvidia0
crw-rw-rw- 1 root root   195,   1 Jun  1 02:56 /dev/nvidia1
crw-rw-rw- 1 root root   195,   2 Jun  1 02:56 /dev/nvidia2
crw-rw-rw- 1 root root   195,   3 Jun  1 02:56 /dev/nvidia3
crw-rw-rw- 1 root root   195,   4 Jun  1 02:56 /dev/nvidia4
crw-rw-rw- 1 root root   195,   5 Jun  1 02:56 /dev/nvidia5
crw-rw-rw- 1 root root   195,   6 Jun  1 02:56 /dev/nvidia6
crw-rw-rw- 1 root root   195,   7 Jun  1 02:56 /dev/nvidia7
crw-rw-rw- 1 root root   195, 255 Jun  1 02:56 /dev/nvidiactl

/dev/nvidia-caps:
total 0
cr-------- 1 root root 248, 1 Jun  1 02:56 nvidia-cap1
cr--r--r-- 1 root root 248, 2 Jun  1 02:56 nvidia-cap2

</code></pre>
<ol start="2">
<li>
<p>Also checked the CUDA_HOME variable: <code>echo $CUDA_HOME</code> now returns <code>/usr/local/cuda-11.7</code> (similar path to <code>which nvcc</code> that returns <code>/usr/local/cuda-11.7/bin/nvcc</code>)</p>
</li>
<li>
<p>If that’s relevant, I’m running Python 3.9 in a virtual environment created specifically for this.</p>
</li>
</ol>

eqy: <p>Are you using a docker container and with the required arguments such as <code>docker run --gpus all</code>? Otherwise, what is the driver version reported by <code>nvidia-smi</code>?</p>

VictorJouault: <p>I’m not running it in a container, just from a python script.</p>
<p>I also copied the output of the <code>nvidia-smi</code> command above, the driver version seems to be <code>515.105.01</code>.</p>

VictorJouault: <p>I found out the problem. My Python installation, on Linux Amazon, was made through LinuxBrew, and it looks like it wasn’t system-wide. I reinstalled Python using pyenv and it now works like a charm.</p>