Process unexpectedly hangs up in torch.distributed.init_process_group()

wuchiz: <p>I trained my model in two nodes, and then it hangs up in initiation.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/1/f1cffb0c0bd417759bda4dfbd74c9fee92b2b66b.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/f1cffb0c0bd417759bda4dfbd74c9fee92b2b66b" title="截屏2021-02-01 下午10.40.24"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f1cffb0c0bd417759bda4dfbd74c9fee92b2b66b_2_690x233.jpeg" alt="截屏2021-02-01 下午10.40.24" data-base62-sha1="yvaFls5Rl3EHRpDEYanT90F0OdJ" width="690" height="233" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f1cffb0c0bd417759bda4dfbd74c9fee92b2b66b_2_690x233.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/f/1/f1cffb0c0bd417759bda4dfbd74c9fee92b2b66b_2_1035x349.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/1/f1cffb0c0bd417759bda4dfbd74c9fee92b2b66b.jpeg 2x" data-dominant-color="181616"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">截屏2021-02-01 下午10.40.24</span><span class="informations">1214×410 50.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/b/eb7bda44290f96266df429d53a3f751bc36c7bcc.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/eb7bda44290f96266df429d53a3f751bc36c7bcc" title="截屏2021-02-01 下午10.41.37"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/eb7bda44290f96266df429d53a3f751bc36c7bcc_2_690x208.jpeg" alt="截屏2021-02-01 下午10.41.37" data-base62-sha1="xBbyCZYCSU54SNbLDIJjNra7Qlu" width="690" height="208" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/eb7bda44290f96266df429d53a3f751bc36c7bcc_2_690x208.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/eb7bda44290f96266df429d53a3f751bc36c7bcc_2_1035x312.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/b/eb7bda44290f96266df429d53a3f751bc36c7bcc_2_1380x416.jpeg 2x" data-dominant-color="171218"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">截屏2021-02-01 下午10.41.37</span><span class="informations">1960×592 183 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
then I add NCCL_DEBUG=INFO and NCCL_DEBUG_SUBSYS=all to see what’s going on, but there is no output file.</p>

mrshenli: <p>Was there any error message? Does it behave differently if you replace <code>gpu10</code> with its IP address?</p>
<blockquote>
<p>then I add NCCL_DEBUG=INFO and NCCL_DEBUG_SUBSYS=all to see what’s going on, but there is no output file.</p>
</blockquote>
<p>It’s possible that <code>init_process_group</code> failed at rendezvous stage (process ip/port discovery using the master as a leader), so that it has not reached NCCL code yet.</p>

wuchiz: <p>no differences. and it just remind me of connect() time out. But when I trained it on a single node with 2 gpus using CUDA_VISIBLE_DEVICES=0,1  python -m torch.distributed.launch --nproc_per_node=2 alphachem_main.py, it works well.</p>

mrshenli: <aside class="quote no-group" data-username="wuchiz" data-post="3" data-topic="110571">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/w/ecae2f/48.png" class="avatar"> wuchiz:</div>
<blockquote>
<p>But when I trained it on a single node with 2 gpus using CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node=2 alphachem_main.py, it works well.</p>
</blockquote>
</aside>
<p>This probably means the two machines cannot talk to each other using the given configuration. Have you tried setting <a href="https://pytorch.org/docs/stable/distributed.html#choosing-the-network-interface-to-use" rel="noopener nofollow ugc"><code>NCCL_SOCKET_IFNAME</code></a> to point to the correct NIC?</p>

wuchiz: <p>It still hangs up in initiation step after setting this environment variable to eth0<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/7/37bbff4a481f8c46dc14f348a4d9442496259a0a.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/37bbff4a481f8c46dc14f348a4d9442496259a0a" title="截屏2021-02-03 上午7.20.06"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37bbff4a481f8c46dc14f348a4d9442496259a0a_2_690x204.jpeg" alt="截屏2021-02-03 上午7.20.06" data-base62-sha1="7X32FAiMnA2JfQU6fFfcDiu45YK" width="690" height="204" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37bbff4a481f8c46dc14f348a4d9442496259a0a_2_690x204.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37bbff4a481f8c46dc14f348a4d9442496259a0a_2_1035x306.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/3/7/37bbff4a481f8c46dc14f348a4d9442496259a0a_2_1380x408.jpeg 2x" data-dominant-color="161319"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">截屏2021-02-03 上午7.20.06</span><span class="informations">2158×640 219 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

wuchiz: <p>I know what’s wrong. the name network interface is eno1 instead of eth0. However it still hangs up in this step. the error is shown in the following figure.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b37ee64f5a1dd5346ddbc57d030004d74068f434.png" data-download-href="https://discuss.pytorch.org/uploads/default/b37ee64f5a1dd5346ddbc57d030004d74068f434" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b37ee64f5a1dd5346ddbc57d030004d74068f434.png" alt="image" data-base62-sha1="pBTlBs6wdZpGPd3R058hT3u9eXW" width="690" height="218" data-dominant-color="1F1F1F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1879×596 44.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>