ProcessGroupNCCL can not find GPUs

Erica_Zheng: <p>Dear All,</p>
<p>I run into<br>
<code>RuntimeError: ProcessGroupNCCL is only supported with GPUs, no GPUs found!</code><br>
But as I check pytorch, it shows I could found my two GPUs.  Please advice. Thank you</p>
<pre><code class="lang-auto">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print(torch.__version__)
2.0.0+cu117
&gt;&gt;&gt; print(torch.cuda.device_count())
2
</code></pre>

ptrblck: <p>Are you setting <code>CUDA_VISIBLE_DEVICES</code> to an invalid value, which would mask the available devices?</p>

hes95075: <p>Hello，I have already set it：&gt;&gt;&gt; torch.cuda.get_device_name(0)<br>
‘NVIDIA GeForce GTX 1650’<br>
but it is stll wrong:<br>
[2024-06-08 09:51:42.677887] [start training example_text_completion.py(16852455) on jd-i0302-dl for huangyongfeng]<br>
[2024-06-08 09:51:42.789996] [训练前检查] 检查[jd-i0302-dl] cpu memory 21.0 &lt; 256G and total gpu memory 0 &lt; 100M 通过<br>
[2024-06-08 09:51:42.900271] found [new_lla3] from [huangyongfeng] in [/hf_shared/hfai_envs/huangyongfeng/new_lla3_0], start loading…<br>
[2024-06-08 09:51:44.338191] user haienv [new_lla3] loaded<br>
[2024-06-08 09:51:50.584100] [W CUDAFunctions.cpp:108] Warning: CUDA initialization: The NVIDIA driver on your system is too old (found version 11040). Please update your GPU driver by downloading and installing a new version from the URL: <a href="http://www.nvidia.com/Download/index.aspx" class="inline-onebox" rel="noopener nofollow ugc">Official Drivers | NVIDIA</a> Alternatively, go to: <a href="https://pytorch.org" rel="noopener nofollow ugc">https://pytorch.org</a> to install a PyTorch version that has been compiled with your version of the CUDA driver. (function operator())<br>
[2024-06-08 09:51:50.585106] Traceback (most recent call last):<br>
[2024-06-08 09:51:50.585128]   File “example_text_completion.py”, line 64, in <br>
[2024-06-08 09:51:50.585214]     fire.Fire(main)<br>
[2024-06-08 09:51:50.585228]   File “/hf_shared/hfai_envs/huangyongfeng/new_lla3_0/lib/python3.8/site-packages/fire/core.py”, line 143, in Fire<br>
[2024-06-08 09:51:50.586689]     component_trace = _Fire(component, args, parsed_flag_args, context, name)<br>
[2024-06-08 09:51:50.586697]   File “/hf_shared/hfai_envs/huangyongfeng/new_lla3_0/lib/python3.8/site-packages/fire/core.py”, line 477, in _Fire<br>
[2024-06-08 09:51:50.587939]     component, remaining_args = _CallAndUpdateTrace(<br>
[2024-06-08 09:51:50.587950]   File “/hf_shared/hfai_envs/huangyongfeng/new_lla3_0/lib/python3.8/site-packages/fire/core.py”, line 693, in _CallAndUpdateTrace<br>
[2024-06-08 09:51:50.588071]     component = fn(*varargs, **kwargs)<br>
[2024-06-08 09:51:50.588078]   File “example_text_completion.py”, line 27, in main<br>
[2024-06-08 09:51:50.588135]     generator = Llama.build(<br>
[2024-06-08 09:51:50.588147]   File “/home/hsr/hfai/llama3-main/llama3-main/llama/generation.py”, line 68, in build<br>
[2024-06-08 09:51:50.589332]     torch.distributed.init_process_group(“nccl”)<br>
[2024-06-08 09:51:50.589339]   File “/hf_shared/hfai_envs/huangyongfeng/new_lla3_0/lib/python3.8/site-packages/torch/distributed/c10d_logger.py”, line 75, in wrapper<br>
[2024-06-08 09:51:50.590771]     return func(*args, **kwargs)<br>
[2024-06-08 09:51:50.590778]   File “/hf_shared/hfai_envs/huangyongfeng/new_lla3_0/lib/python3.8/site-packages/torch/distributed/c10d_logger.py”, line 89, in wrapper<br>
[2024-06-08 09:51:50.590829]     func_return = func(*args, **kwargs)<br>
[2024-06-08 09:51:50.590835]   File “/hf_shared/hfai_envs/huangyongfeng/new_lla3_0/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 1312, in init_process_group<br>
[2024-06-08 09:51:50.592212]     default_pg, _ = _new_process_group_helper(<br>
[2024-06-08 09:51:50.592219]   File “/hf_shared/hfai_envs/huangyongfeng/new_lla3_0/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 1533, in _new_process_group_helper<br>
[2024-06-08 09:51:50.592433]     backend_class = ProcessGroupNCCL(<br>
[2024-06-08 09:51:50.592439] ValueError: ProcessGroupNCCL is only supported with GPUs, no GPUs found!<br>
Is the qustion about “The NVIDIA driver on your system is too old”?  but , I run this programm in WSL,and if I update my NVIDIA version to the newst one , it seems not avilable " segemantation fault". How can I address it?</p>

ptrblck: <aside class="quote no-group" data-username="hes95075" data-post="3" data-topic="176794">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hes95075/48/70595_2.png" class="avatar"> hes95075:</div>
<blockquote>
<p>Warning: CUDA initialization: The NVIDIA driver on your system is too old (found version 11040). Please update your GPU driver by downloading and installing a new version from the URL: <a href="http://www.nvidia.com/Download/index.aspx">Official Drivers | NVIDIA</a> Alternatively, go to: <a href="https://pytorch.org">https://pytorch.org</a> to install a PyTorch version that has been compiled with your version of the CUDA driver.</p>
</blockquote>
</aside>
<p>You would need to install a newer driver or install a PyTorch binary with a compatible CUDA version (e.g. with CUDA 11.8 in case you are using a driver shipped with CUDA 11).</p>

hes95075: <p>Thank you, but I have already reinstalled CUDA and PyTorch. Their versions are 12.1, and my NVIDIA driver is 12.4. However, I’m still encountering the same problem. – [W CUDAFunctions.cpp:108] Warning: CUDA initialization: The NVIDIA driver on your system is too old (found version 11040). Please update your GPU driver by downloading and installing a new version from the URL: <a href="http://www.nvidia.com/Download/index.aspx" class="inline-onebox" rel="noopener nofollow ugc">Official Drivers | NVIDIA</a> Alternatively, go to: <a href="https://pytorch.org" rel="noopener nofollow ugc">https://pytorch.org</a> to install a PyTorch version that has been compiled with your version of the CUDA driver. (function operator())</p>

ptrblck: <aside class="quote no-group" data-username="hes95075" data-post="5" data-topic="176794">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hes95075/48/70595_2.png" class="avatar"> hes95075:</div>
<blockquote>
<p>and my NVIDIA driver is 12.4.</p>
</blockquote>
</aside>
<p>The error message points to a driver shipped with CUDA 11.4, so make sure a single driver is installed, update it to a CUDA 12 driver, or install the PyTorch binaries with CUDA 11.8.</p>

hes95075: <p>Thank you for your help，I have already addressed this problem~</p>