GPUs being detected but "RuntimeError: ProcessGroupNCCL is only supported with GPUs, no GPUs found?" HELP!

confusedpotato: <p>Hello,</p>
<p>I am trying to recreate the results of this study: <a href="https://github.com/insitro/ChannelViT/tree/main" class="inline-onebox" rel="noopener nofollow ugc">GitHub - insitro/ChannelViT: Channel Vision Transformers: An Image Is Worth C x 16 x 16 Words</a></p>
<p>However, I keep running into the error</p>
<blockquote>
<p>RuntimeError: ProcessGroupNCCL is only supported with GPUs, no GPUs found?</p>
</blockquote>
<p>The device I am using is from a GPU sever with a Titan RTX and  GTX 1080 and according to using <code>nvidia-smi</code> its with CUDA 10.1.</p>
<p>I can’t seem to figure out why the GPUs aren’t being detected. They are being detected when running <code>print(torch.cuda.device_count())</code> as 2 GPUs and same with running <code>print(os.environ["CUDA_VISIBLE_DEVICES"])</code></p>
<p>I am using pytorch 2.01 with CUDA 11.7, could this be an issue with the mismatched CUDA version installed on to the server? I have tried running it with using only 1 of the GPUs but same error comes up.</p>
<p>I’ve been told that I am able to use conda to isolate the CUDA version rather than managing it globally. Maybe I’ve done something wrong in the installation, but I can’t seem to figure it out!</p>
<p>Please let me know what I can do!</p>

confusedpotato: <p>I forgot to mention that the pytorch log also comes out as detecting the GPUs</p>
<blockquote>
<p>GPU available: True (cuda), used: True<br>
TPU available: False, using: 0 TPU cores<br>
IPU available: False, using: 0 IPUs<br>
HPU available: False, using: 0 HPUs</p>
</blockquote>

ptrblck: <aside class="quote no-group" data-username="confusedpotato" data-post="1" data-topic="195731">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/7ba0ec/48.png" class="avatar"> confusedpotato:</div>
<blockquote>
<p>and according to using <code>nvidia-smi</code> its with CUDA 10.1.</p>
</blockquote>
</aside>
<p>Your NVIDIA driver might be too old and you might need to update it in order to run PyTorch with CUDA 11.7.</p>

confusedpotato: <p>Yea, that’s what I assume. Is there a possible different solution to this? This is a GPU server organized by my organization and I am not sure if this is something they can update.</p>
<p>Thanks for the quick respond though!</p>

ptrblck: <p>If you cannot update the NVIDIA driver you would need to use older an older CUDA toolkit and thus also downgrade PyTorch to a version shipping with CUDA 10, which is quite old by now.</p>

confusedpotato: <p>Hi there. Just wondering, if the CUDA driver is 11.0 and driver is 450.119 (I forgot the exact number) would that be able to run pytorch 2.0?</p>

ptrblck: <p>Yes, if you install the PyTorch binaries with CUDA 11.8 as the min. NVIDIA driver would be &gt;=450.80.02 on Linux. To run the PyTorch binaries with CUDA 12.1U1 you would need to install NVIDIA driver &gt;=525.60.13 as seen in the <a href="https://docs.nvidia.com/deploy/cuda-compatibility/">compatibility matrix</a>.</p>

sree_lakshmi_narayan: <p>Hi,</p>
<p>After executing the command to check the availablity of GPU.</p>
<p>python3 -c “import torch; print([(i, torch.cuda.get_device_properties(i)) for i in range(torch.cuda.device_count())])”</p>
<p>output:</p>
<p>[(0, _CudaDeviceProperties(name=‘Tesla V100-SXM2-16GB’, major=7, minor=0, total_memory=16160MB, multi_processor_count=80))]</p>
<p>But still getting error</p>
<p>RuntimeError: ProcessGroupNCCL is only supported with GPUs, no GPUs found!<br>
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 17848) of binary:</p>

ptrblck: <p>Could you post a minimal and executable code snippet reproducing the issue?</p>

guotong1988: <pre><code class="lang-auto">CUDA Version: 11.4 
torch==2.0.0
</code></pre>
<p>works</p>
<pre><code class="lang-auto">CUDA Version: 11.4 
torch==2.1.0 / torch==2.2.0 / torch==2.3.0
</code></pre>
<p>not works</p>