Torch.cuda.device_count() is 0

coppock: <p>When I set the CUDA_VISIBLE_DEVICES environment variable to the UUID of one of my GPUs, torch doesn’t enumerate the device. However, using the device index works fine.</p>
<pre><code class="lang-auto">$ nvidia-smi -L
GPU 0: NVIDIA A30 (UUID: GPU-28c5c635-0214-457e-4d24-0282dac9957d)
GPU 1: NVIDIA A30 (UUID: GPU-e031b04a-4f36-83a9-373b-0d6f231a87dc)
$ python
Python 3.8.10 (default, Nov 14 2022, 12:59:47) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import os
&gt;&gt;&gt; os.environ['CUDA_VISIBLE_DEVICES'] = 'GPU-28c5c635-0214-457e-4d24-0282dac9957d'
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.cuda.is_available()
True
&gt;&gt;&gt; torch.cuda.device_count()
0
&gt;&gt;&gt; torch._C._cuda_getDeviceCount()
1
$ python
&gt;&gt;&gt; import os
&gt;&gt;&gt; os.environ['CUDA_VISIBLE_DEVICES'] = '0'
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.cuda.is_available()
True
&gt;&gt;&gt; torch.cuda.device_count()
1
&gt;&gt;&gt; 
</code></pre>
<p>Interestingly enough, torch.cuda.device_count() isn’t consistent with torch._C._cuda_getDeviceCount() when using the UUID.</p>
<p>The reason why this is an issue is that I’m inferencing on PyTorch on a MIG-partitioned GPU, and I need to give the script a single MIG slice. As far as I am aware, UUIDs are the only way to do this. When running the model, PyTorch raises <code>RuntimeError: Attempting to deserialize object on CUDA device 0 but torch.cuda.device_count() is 0. Please use torch.load with map_location to map your storages to an existing device.</code></p>
<p>Is this a PyTorch bug? If so, I’ll open a GitHub issue?</p>

coppock: <p>Interestingly enough, on Docker (using the <code>--gpus device=</code> flag, I can specify the UUID without the above-described issue.</p>

ptrblck: <p>Specifying the UUID sounds like the right approach and was working for me in the past using MIG. However, I usually <code>export</code> the env variable in my terminal so let me double check your code snippet which sets it inside the script.</p>

xenova: <p>I’m having the exact same issue (and also on a MIG-partitioned GPU). I didn’t have this issue on a previous version of pytorch though (can’t remember which… just know that it worked in the past)</p>
<p>Do you have any updates or ways to fix it?</p>

coppock: <p>No, I haven’t fixed the issue. I can update this thread with versions I’m using, though. The problem exists on Python 3.8.10 with torch 1.13.0+cu117 on an Ubuntu 20 machine with NVIDIA driver 520.61.05 and CUDA 11.8. The Docker image that allowed me to use the UUID through the <code>--gpus</code> flag was <code>pytorch/pytorch:1.13.1-cuda11.6-cudnn8-runtime</code> which has Python 3.10.8 and torch 1.13.0.</p>

xenova: <p>After struggling with this for the past few days, the only “solution” I have found is to downgrade to earlier versions of pytorch.</p>
<p>Adding this to my requirements.txt file seems to have fixed the issue.</p>
<pre><code class="lang-auto">--extra-index-url https://download.pytorch.org/whl/cu111
torch==1.9.0+cu111
torchvision==0.10.0+cu111
</code></pre>
<p>Unfortunately, this is a massive step back from the current versions of the respective programs, but it was the only thing that worked.</p>
<p>If you haven’t already, could you (<a class="mention" href="/u/coppock">@coppock</a>) open a GitHub issue? Seems like it is a bug with newer pytorch versions.</p>

coppock: <p>Very good. Looks like there’s already an <a href="https://github.com/pytorch/pytorch/issues/90543" rel="noopener nofollow ugc">issue</a> open for two weeks. Should’ve checked earlier!</p>