Init_process_group with launch.py --nnode=2 hangs always in all machines

WurmD: <p><strong>Minimal example code:</strong></p>
<pre><code class="lang-auto">import os
import torch.distributed as dist

os.environ['MASTER_ADDR'] = '148.251.86.243' # My master server IP
print(f"[ {os.getpid()} ] Initializing process group")
dist.init_process_group(backend="nccl")
print(f"[ {os.getpid()} ] world_size = {dist.get_world_size()}, " + f"rank = {dist.get_rank()}, backend={dist.get_backend()}")
</code></pre>
<ul>
<li>
<p>The master address is hardcoded to my server’s IP: <code>os.environ['MASTER_ADDR'] = '148.251.86.243'</code></p>
</li>
<li>
<p>Launching with launch.py in master server hangs:</p>
</li>
</ul>
<pre><code class="lang-auto">$ export NCCL_DEBUG=INFO ; export NCCL_DEBUG_SUBSYS=ALL ; python3 /home/dario/.local/lib/python3.6/site-packages/torch/distributed/launch.py --nnode=2 --node_rank=0 --nproc_per_node=1 msg3d/MSG3D/tuturial_DistributedDataParallel.py --local_world_size=1
[26236] Initializing process group with: {'MASTER_ADDR': '148.251.86.243', 'MASTER_PORT': '29500', 'RANK': '0', 'WORLD_SIZE': '2'}
</code></pre>
<ul>
<li>And in second server it also hangs:</li>
</ul>
<pre><code class="lang-auto">$ export NCCL_DEBUG=INFO ; export NCCL_DEBUG_SUBSYS=ALL ; python3 /home/dario/.local/lib/python3.6/site-packages/torch/distributed/launch.py --nnode=2 --node_rank=1 --nproc_per_node=1 msg3d/MSG3D/tuturial_DistributedDataParallel.py --local_world_size=1
[17186] Initializing process group with: {'MASTER_ADDR': '148.251.86.243', 'MASTER_PORT': '29500', 'RANK': '1', 'WORLD_SIZE': '2'}
</code></pre>
<ul>
<li><a href="https://discuss.pytorch.org/t/pytorch-distributed-example-code-hang-deadlock/83335">No difference between <code>dist.init_process_group(backend="nccl")</code> and <code>"gloo"</code></a></li>
<li><a href="https://discuss.pytorch.org/t/torch-distributed-launch-hanged/112223">No difference when setting <code>os.environ['NCCL_SOCKET_IFNAME'] = 'lo'</code></a></li>
<li><a href="https://pytorch.org/docs/stable/distributed.html#other-nccl-environment-variables" rel="noopener nofollow ugc">Setting <code>export NCCL_DEBUG=INFO ;    export NCCL_DEBUG_SUBSYS=ALL ;</code> produces no new output</a></li>
<li>No difference between PyTorch 1.7.1 and 1.8.0-rc4</li>
<li><a href="https://github.com/pytorch/pytorch/issues/23074#issuecomment-513445542" rel="noopener nofollow ugc">dist.init_process_group(backend=“nccl”, <strong>init_method=‘env://’</strong>) didn’t help</a></li>
</ul>
<p>What else should I try?</p>
<p><strong>Environment:</strong></p>
<pre><code class="lang-auto">PyTorch version: 1.7.1 or 1.8.0.dev20210208+cu110
Is debug build: True
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.17.1

Python version: 3.6 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 11.2.142
GPU models and configuration: GPU 0: GeForce GTX 1080
Nvidia driver version: 460.32.03
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.18.4
[pip3] torch==1.7.1 or 1.8.0.dev20210208+cu110
[pip3] torchvision==0.8.1
[conda] Could not collect
</code></pre>

pritamdamania87: <p>It looks like you’re using two processes but setting WORLD_SIZE to 1, I think setting WORLD_SIZE to 2 should fix this issue.</p>

WurmD: <p>Good spot!</p>
<p>But now it hangs on both =/, editted above</p>
<p><code>$ python3 /home/dario/.local/lib/python3.6/site-packages/torch/distributed/launch.py --nnode=2 --node_rank=0 --nproc_per_node=1 msg3d/MSG3D/tuturial_DistributedDataParallel.py --local_world_size=1</code></p>
<p><code>[23616] Initializing process group with: {'MASTER_ADDR': '148.251.86.243', 'MASTER_PORT': '29500', 'RANK': '0', 'WORLD_SIZE': '2'}</code></p>

WurmD: <p>Continuing to try debug this</p>
<ul>
<li>Setting <code>export NCCL_DEBUG=INFO ;    export NCCL_DEBUG_SUBSYS=ALL ;</code> produces no new output</li>
</ul>
<p>Editted in above</p>

WurmD: <p>Next attempt:<br>
<a href="https://download.pytorch.org/whl/nightly/cu110/torch-1.8.0.dev20210208%2Bcu110-cp36-cp36m-linux_x86_64.whl" rel="noopener nofollow ugc">torch-1.8.0.dev20210208%2Bcu110-cp36-cp36m-linux_x86_64.whl</a></p>
<p>With<br>
<code>backend="nccl"</code><br>
<code>NCCL_DEBUG: INFO</code><br>
<code>NCCL_DEBUG_SUBSYS: ALL</code><br>
Absolutely no change, <code>dist.init_process_group(backend="nccl")</code> hangs in both machines and no extra output is given</p>

WurmD: <p>Deleted down to a minimal example:</p>
<pre><code class="lang-auto">import os
import torch.distributed as dist

os.environ['MASTER_ADDR'] = '148.251.86.243'
print(f"[ {os.getpid()} ] Initializing process group")
dist.init_process_group(backend="nccl")
print(f"[ {os.getpid()} ] world_size = {dist.get_world_size()}, " + f"rank = {dist.get_rank()}, backend={dist.get_backend()}")
</code></pre>
<p>editted above</p>

WurmD: <ul>
<li>dist.init_process_group(backend=“nccl”, <strong>init_method=‘env://’</strong>) didn’t help</li>
</ul>

WurmD: <p>Created issue <a href="https://github.com/pytorch/pytorch/issues/52848" class="inline-onebox" rel="noopener nofollow ugc">init_process_group with launch.py --nnode=2 hangs always in all machines · Issue #52848 · pytorch/pytorch · GitHub</a></p>

WurmD: <p>Ok, managed to open a port (29500) that allows init_process_group to run correctly.</p>
<p>In my firewall settings it looks like</p>
<pre><code class="lang-auto">Source IP | Destination IP | Source port | Destination port | Protocol | TCP flags | Action 
          |                |             |            29500 |      tcp |           | accept
</code></pre>
<p><strong>After</strong> master process is running (that starts listening on 29500), then doing <code>telnet 148.251.86.243 29500</code> says</p>
<pre><code class="lang-auto">Trying 148.251.86.243...
Connected to gpu14.
Escape character is '^]'.
</code></pre>
<p>confirming the port is correctly open.</p>
<p>And running the minimal example above on the second server makes init_process_group return correctly and print my second print above<br>
<code>[ 8172 ] world_size = 2, rank = 0, backend=nccl</code></p>