GPU not available with CUDA 12.1 on Amazon V100 GPU instance

AKM: <p>Hi, I am trying to set up Pytorch on a Amazon V100 GPU instance with CUDA 12.1.</p>
<p>The Pytorch download page (<a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a>) didn’t have a version for CUDA 12.1 so I used:</p>
<pre><code class="lang-auto">conda install pytorch torchvision torchaudio pytorch-cuda=11.7 -c pytorch -c nvidia
</code></pre>
<p>as this discussion suggests: <a href="https://discuss.pytorch.org/t/install-pytorch-with-cuda-12-1/174294" class="inline-onebox">Install pytorch with Cuda 12.1</a></p>
<p>I have the Nvidia drivers configured correctly on my instance:</p>
<pre><code class="lang-auto">% nvidia-smi
Wed Mar 15 16:40:19 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla V100-SXM2-16GB            Off| 00000000:00:1E.0 Off |                    0 |
| N/A   43C    P0               43W / 300W|      0MiB / 16384MiB |      1%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

</code></pre>
<p>and:</p>
<pre><code class="lang-auto">nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Feb__7_19:32:13_PST_2023
Cuda compilation tools, release 12.1, V12.1.66
Build cuda_12.1.r12.1/compiler.32415258_0
</code></pre>
<p>But on installing Pytorch with the above command I get:</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.version.cuda
'11.7'
&gt;&gt;&gt; torch.cuda.is_available()
False
</code></pre>
<p>Any help would be great,<br>
Thanks</p>

ptrblck: <p>Are you able to run any other CUDA application on this system and in your environment?</p>

AKM: <p>Thanks <span class="mention">@ptrblck_de</span>, what would be the simplest way to check this?</p>

AKM: <p>So, I tried to see if the environment can support a pytorch-gpu docker image:</p>
<pre><code class="lang-auto">docker run --gpus all -it --rm nvcr.io/nvidia/pytorch:23.02-py3

Status: Downloaded newer image for nvcr.io/nvidia/pytorch:23.02-py3
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].
</code></pre>
<p>Looks like a CUDA problem?</p>

ptrblck: <p>It looks more like a driver problem. What does <code>nvidia-smi</code> return on the bare metal machine (without running any docker containers)?</p>

AKM: <pre><code class="lang-auto">% nvidia-smi
Wed Mar 15 16:40:19 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.30.02              Driver Version: 530.30.02    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla V100-SXM2-16GB            Off| 00000000:00:1E.0 Off |                    0 |
| N/A   43C    P0               43W / 300W|      0MiB / 16384MiB |      1%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
</code></pre>

ptrblck: <p>Could you check if the correct <code>nvidia-container-toolkit</code> is installed?<br>
You could take a look at <a href="https://github.com/NVIDIA/nvidia-docker/issues/1243">this issue</a> which describes the same issue and fixes.</p>