Unable to executed example.py in examples/distributed/ddp across nodes?

nsriniva03: <p>Hello,</p>
<p>I am trying to test out distributed training across nodes using the example.py script provided in examples/distributed/ddp.</p>
<p>The code is not executed beyond  dist.init_process_group(backend=“nccl”) statement and just hangs.</p>
<p>I have used the following command to run the code.</p>
<p>On Node 1:<br>
export NCCL_DEBUG=INFO ; export NCCL_DEBUG_SUBSYS=ALL; python /home/nisha/anaconda3/envs/pytorch1.8_env/lib/python3.8/site-packages/torch/distributed/launch.py --nnode=2 --node_rank=0 --nproc_per_node=8 --master_addr=162.157.89.143 --master_port=20051 example.py --local_world_size=8</p>
<p>Output Node 1:</p>
<p>[17182] Initializing process group with: {‘MASTER_ADDR’: ‘162.157.89.143’, ‘MASTER_PORT’: ‘20051’, ‘RANK’: ‘2’, ‘WORLD_SIZE’: ‘16’}<br>
[17180] Initializing process group with: {‘MASTER_ADDR’: ‘162.157.89.143’, ‘MASTER_PORT’: ‘20051’, ‘RANK’: ‘0’, ‘WORLD_SIZE’: ‘16’}<br>
[17183] Initializing process group with: {‘MASTER_ADDR’: ‘162.157.89.143’, ‘MASTER_PORT’: ‘20051’, ‘RANK’: ‘3’, ‘WORLD_SIZE’: ‘16’}<br>
[17184] Initializing process group with: {‘MASTER_ADDR’: ‘162.157.89.143’, ‘MASTER_PORT’: ‘20051’, ‘RANK’: ‘4’, ‘WORLD_SIZE’: ‘16’}<br>
[17186] Initializing process group with: {‘MASTER_ADDR’: ‘162.157.89.143’, ‘MASTER_PORT’: ‘20051’, ‘RANK’: ‘6’, ‘WORLD_SIZE’: ‘16’}<br>
[17187] Initializing process group with: {‘MASTER_ADDR’: ‘162.157.89.143’, ‘MASTER_PORT’: ‘20051’, ‘RANK’: ‘7’, ‘WORLD_SIZE’: ‘16’}<br>
[17181] Initializing process group with: {‘MASTER_ADDR’: ‘162.157.89.143’, ‘MASTER_PORT’: ‘20051’, ‘RANK’: ‘1’, ‘WORLD_SIZE’: ‘16’}<br>
[17185] Initializing process group with: {‘MASTER_ADDR’: ‘162.157.89.143’, ‘MASTER_PORT’: ‘20051’, ‘RANK’: ‘5’, ‘WORLD_SIZE’: ‘16’}</p>
<p>On Node 2:<br>
export NCCL_DEBUG=INFO ; export NCCL_DEBUG_SUBSYS=ALL; python /home/nisha/anaconda3/envs/pytorch1.8_env/lib/python3.8/site-packages/torch/distributed/launch.py --nnode=2 --node_rank=1 --nproc_per_node=8 --master_addr=162.157.89.143 --master_port=20051 example.py --local_world_size=8</p>
<p>Output on node 2:</p>
<p>[36854] Initializing process group with: {‘MASTER_ADDR’: ‘162.157.89.143’, ‘MASTER_PORT’: ‘20051’, ‘RANK’: ‘15’, ‘WORLD_SIZE’: ‘16’}<br>
[36851] Initializing process group with: {‘MASTER_ADDR’: ‘162.157.89.143’, ‘MASTER_PORT’: ‘20051’, ‘RANK’: ‘12’, ‘WORLD_SIZE’: ‘16’}<br>
[36847] Initializing process group with: {‘MASTER_ADDR’: ‘162.157.89.143’, ‘MASTER_PORT’: ‘20051’, ‘RANK’: ‘8’, ‘WORLD_SIZE’: ‘16’}<br>
[36848] Initializing process group with: {‘MASTER_ADDR’: ‘162.157.89.143’, ‘MASTER_PORT’: ‘20051’, ‘RANK’: ‘9’, ‘WORLD_SIZE’: ‘16’}<br>
[36853] Initializing process group with: {‘MASTER_ADDR’: ‘162.157.89.143’, ‘MASTER_PORT’: ‘20051’, ‘RANK’: ‘14’, ‘WORLD_SIZE’: ‘16’}<br>
[36852] Initializing process group with: {‘MASTER_ADDR’: ‘162.157.89.143’, ‘MASTER_PORT’: ‘20051’, ‘RANK’: ‘13’, ‘WORLD_SIZE’: ‘16’}<br>
[36850] Initializing process group with: {‘MASTER_ADDR’: ‘162.157.89.143’, ‘MASTER_PORT’: ‘20051’, ‘RANK’: ‘11’, ‘WORLD_SIZE’: ‘16’}<br>
[36849] Initializing process group with: {‘MASTER_ADDR’: ‘162.157.89.143’, ‘MASTER_PORT’: ‘20051’, ‘RANK’: ‘10’, ‘WORLD_SIZE’: ‘16’}</p>
<hr>
<p>I have tested that the port is open and communicating between the machines by starting the code on Node 1 and running telnet 162.157.89.143 20051 on Node 2.</p>
<p>I am not seeing at NCCL report although I have included NCLL_DEBUG. The program just hangs and nothing else happens.</p>
<p>I also tried to attach gdb to both the processes,</p>
<pre><code class="lang-auto">gdb -p &lt;pid&gt;
set logging on
thread apply all bt
# Output will be in gdb.txt file
</code></pre>
<p>And gdb.txt was empty.</p>
<p>Another strange behavior is that, when I launch the script, the processes started are not shown on nvidia-smi.</p>
<p>And suggestions on things I could try?</p>
<p>Thank you</p>