RuntimeError: ProcessGroupNCCL is only supported with GPUs, no GPUs found!

dmlpt: <p>Hi All,<br>
I am trying to run <a href="https://github.com/facebookresearch/dino" rel="noopener nofollow ugc">DINO </a> on multiple nodes with facebookincubator/submitit repo. We have a slurm server and I am able to train DINO on the slurm server using a single node (8gpus) [WITHOUT USING submitit] but when I try to run with multiple nodes, I am getting the below error:</p>
<p>submitit ERROR (2021-07-30 01:10:30,581) - Submitted job triggered an exception<br>
Traceback (most recent call last):<br>
File “/home/user/skanaconda3/envs/url/lib/python3.8/runpy.py”, line 194, in _run_module_as_main<br>
return _run_code(code, main_globals, None,<br>
File “/home/user/skanaconda3/envs/url/lib/python3.8/runpy.py”, line 87, in _run_code<br>
exec(code, run_globals)<br>
File “/home/user/skanaconda3/envs/url/lib/python3.8/site-packages/submitit/core/_submit.py”, line 11, in<br>
submitit_main()<br>
File “/home/user/skanaconda3/envs/url/lib/python3.8/site-packages/submitit/core/submission.py”, line 71, in submitit_main<br>
process_job(args.folder)<br>
File “/home/user/skanaconda3/envs/url/lib/python3.8/site-packages/submitit/core/submission.py”, line 64, in process_job<br>
raise error<br>
File “/home/user/skanaconda3/envs/url/lib/python3.8/site-packages/submitit/core/submission.py”, line 53, in process_job<br>
result = delayed.result()<br>
File “/home/user/skanaconda3/envs/url/lib/python3.8/site-packages/submitit/core/utils.py”, line 128, in result<br>
self._result = self.function(*self.args, **self.kwargs)<br>
File “run_with_submitit.py”, line 67, in <strong>call</strong><br>
main_dino_initialize_all.train_dino(self.args)<br>
File “/home/user/code/dino/main_dino_initialize_all.py”, line 143, in train_dino<br>
utils.init_distributed_mode(args)<br>
File “/home/user/code/dino/utils.py”, line 468, in init_distributed_mode<br>
dist.init_process_group(<br>
File “/home/user/skanaconda3/envs/url/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 439, in init_process_group<br>
_default_pg = _new_process_group_helper(<br>
File “/home/user/skanaconda3/envs/url/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py”, line 528, in _new_process_group_helper<br>
pg = ProcessGroupNCCL(<br>
RuntimeError: ProcessGroupNCCL is only supported with GPUs, no GPUs found!</p>
<p>From logs, I see that the job initially gets assigned to two nodes [with 8 gpus in each node] and then stops with the above error. I think the code crashes at <a href="https://github.com/pytorch/pytorch/blob/1db2d9b0a81d7092485599892ed238053bdd8cec/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp#L454" rel="noopener nofollow ugc">this line</a> . Why does at::cuda::getNumGPUs() returns 0 when there are gpus available?</p>
<p>Thanks in advance!</p>

H-Huang: <p>I am not familiar with submitit so I am unsure of how to validate the number of GPUs that is using. Before <code>init_process_group</code> can you also try printing the value of <code>torch.cuda.device_count()</code>? (<a href="https://pytorch.org/docs/master/generated/torch.cuda.device_count.html?highlight=device_count" class="inline-onebox" rel="noopener nofollow ugc">torch.cuda.device_count — PyTorch master documentation</a>). This may help to narrow down why there are any GPUs detected and whether this is an issue in the distributed package.</p>