The ddp seem to be disable to find the second node

wuchiz: <p>I train my model in two nodes(4 gpus) with ddp.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/9/591972c171c6b7a2d7b501e0367bd7431f434fdb.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/591972c171c6b7a2d7b501e0367bd7431f434fdb" title="截屏2021-02-05 下午5.18.47"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/591972c171c6b7a2d7b501e0367bd7431f434fdb_2_690x287.jpeg" alt="截屏2021-02-05 下午5.18.47" data-base62-sha1="cId15mBE3LzhZm5Da76hXbuHb19" width="690" height="287" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/9/591972c171c6b7a2d7b501e0367bd7431f434fdb_2_690x287.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/5/9/591972c171c6b7a2d7b501e0367bd7431f434fdb.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/9/591972c171c6b7a2d7b501e0367bd7431f434fdb.jpeg 2x" data-dominant-color="131117"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">截屏2021-02-05 下午5.18.47</span><span class="informations">1022×426 94 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/e/ce4ff49f1adcf618e05ded2ce7ca50c1e90705b1.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/ce4ff49f1adcf618e05ded2ce7ca50c1e90705b1" title="截屏2021-02-05 下午5.27.42"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/e/ce4ff49f1adcf618e05ded2ce7ca50c1e90705b1_2_690x339.jpeg" alt="截屏2021-02-05 下午5.27.42" data-base62-sha1="tr7F6afM8ZygF2tHLBzvHXTTMCB" width="690" height="339" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/e/ce4ff49f1adcf618e05ded2ce7ca50c1e90705b1_2_690x339.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/c/e/ce4ff49f1adcf618e05ded2ce7ca50c1e90705b1.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/e/ce4ff49f1adcf618e05ded2ce7ca50c1e90705b1.jpeg 2x" data-dominant-color="131215"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">截屏2021-02-05 下午5.27.42</span><span class="informations">772×380 48 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
When I log in the first node, it seems to functions well. when I use ps aux|grep python. there are two tasks running<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/a/5a2711a86813c4e6ba2a9a4c72db6ef584ceb20e.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5a2711a86813c4e6ba2a9a4c72db6ef584ceb20e" title="截屏2021-02-05 下午5.23.31"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a2711a86813c4e6ba2a9a4c72db6ef584ceb20e_2_690x39.jpeg" alt="截屏2021-02-05 下午5.23.31" data-base62-sha1="cRwFZi8ncswuTtq0MTJ3i62CjHE" width="690" height="39" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a2711a86813c4e6ba2a9a4c72db6ef584ceb20e_2_690x39.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a2711a86813c4e6ba2a9a4c72db6ef584ceb20e_2_1035x58.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/a/5a2711a86813c4e6ba2a9a4c72db6ef584ceb20e_2_1380x78.jpeg 2x" data-dominant-color="1D1C1C"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">截屏2021-02-05 下午5.23.31</span><span class="informations">2874×164 98.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
but when I log in the second node, there are no any tasks running<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1c88567616741c0ec13df52cd778435bb24f9bb7.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/1c88567616741c0ec13df52cd778435bb24f9bb7" title="截屏2021-02-05 下午5.21.48"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c88567616741c0ec13df52cd778435bb24f9bb7_2_690x36.jpeg" alt="截屏2021-02-05 下午5.21.48" data-base62-sha1="44ptebrZqTN8twrKmeKs29N2Ffh" width="690" height="36" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c88567616741c0ec13df52cd778435bb24f9bb7_2_690x36.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c88567616741c0ec13df52cd778435bb24f9bb7_2_1035x54.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/c/1c88567616741c0ec13df52cd778435bb24f9bb7_2_1380x72.jpeg 2x" data-dominant-color="201F1F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">截屏2021-02-05 下午5.21.48</span><span class="informations">1462×78 28.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
so how do the ddp find the second node?</p>

osalpekar: <p>If I understand correctly, you are trying to train with 4 GPUs, 2 on one machine and 2 on another machine? If this is the case, then you will need to launch your training script separately on each machine. The node_rank for launch script on the first machine should be 0 and node_rank passed to the launch script on the second machine should be 1. It seems here like you are passing 2 separate node_ranks for processes launched on the same machine.</p>
<p>See the multi-node multi-process distributed launch example here: <a href="https://pytorch.org/docs/stable/distributed.html#launch-utility" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 1.7.0 documentation</a></p>

wuchiz: <p>thanks! I got it. By the way, is there any ways to automatically do that? manuallying launch the task in each node is not convenient.</p>

osalpekar: <p>We don’t provide a way of doing this natively in PyTorch. Writing a simple bash script to do this should be doable though (take a list of hostnames, ssh into each one, copy over pytorch program, and run). You could also use slurm/mpirun if those are available in your environment.</p>
<p>If you decide to use torchelastic for distributed training, then there are some plugins available to simplify training in the cloud: <a href="https://github.com/pytorch/elastic/tree/master/kubernetes" class="inline-onebox" rel="noopener nofollow ugc">elastic/kubernetes at master · pytorch/elastic · GitHub</a></p>