Connection refused with GLOO process group initialization

username: HuYang719
cooked: <p>Dear all, I attempt to use TCP initial method to initial two nodes in AWS EC2 (different area but with all traffic allow):</p>
<pre data-code-wrap="py"><code class="lang-py">def init_process(rank, size, backend='gloo'):
    """ Initialize the distributed environment. """
    os.environ["TP_SOCKET_IFNAME"] = 'ens5'
    os.environ["GLOO_SOCKET_IFNAME"] = 'ens5'
    initial_tcp = "tcp://44.234.152.249:23456"
    dist.init_process_group(backend, init_method=initial_tcp,rank=rank, world_size=size, timeout=timeout)
    run(rank, size)

if __name__ == "__main__":
    size = 2
    processes = []
    mp.set_start_method("spawn")
    init_process(0, size, backend='gloo') # node 0
# init_process(1, size, backend='gloo') for node 1
</code></pre>
<p>And I make sure these two nodes can communicate with public IP using socket. I tried binding the public IP with the node rank 1:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/0/0009dded264438ad2d6e407ed8dcf99f08ef1f77.png" data-download-href="https://discuss.pytorch.org/uploads/default/0009dded264438ad2d6e407ed8dcf99f08ef1f77" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/0/0009dded264438ad2d6e407ed8dcf99f08ef1f77.png" alt="image" data-base62-sha1="l8F8S9nxlW1N5hSb6bXaxZdS7" width="690" height="122" data-dominant-color="414141"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">755×134 7.41 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>However, when setting up the initialization, the <strong>rank 0</strong> node reports error:</p>
<pre data-code-wrap="sh"><code class="lang-sh">RuntimeError: [/opt/conda/conda-bld/pytorch_1634272172048/work/third_party/gloo/gloo/transport/tcp/pair.cc:799] 
connect [13.234.25.113]:42897: Connection refused
</code></pre>
<p>and rank 1 node (with public IP 13.234.25.113):</p>
<pre data-code-wrap="sh"><code class="lang-sh">RuntimeError: Connection reset by peer
</code></pre>
<p>I have used <code>netcat</code> to make sure that the port 42897 on the rank 1 node can be connected successfully with the rank 0 node.</p>
<p>I do not know why this happens, is it because rank 1 node opened a different port for initialization (that’s my understanding for connection refused). Could anyone help me? Thanks!</p>

username: Yanli_Zhao
cooked: <p><a class="mention" href="/u/huyang719">@HuYang719</a> do you want to try to use the same port for two nodes and see whether it can resolve the issue? also cc <a class="mention" href="/u/cbalioglu">@cbalioglu</a></p>

username: HuYang719
cooked: <p>Hi Yanli, I am not sure how to use the same port to initiate the process group. The worker node seems to try to open a random port and (also changes with the time) send the SYN package to the master node. I feel it is not controlled by the code.<br>
Is there any method to use the same port for both worker node and master node?</p>

username: HuYang719
cooked: <p>Hi, dear <a class="mention" href="/u/yanli_zhao">@Yanli_Zhao</a> <a class="mention" href="/u/cbalioglu">@cbalioglu</a> , I want to update some progress here. But  I wonder the most essential question for torch.distributed (GLOO/NCCL backend):</p>
<ol>
<li>If the master and worker nodes can be accessed with public IP (they may be in their own LAN but no firewall, and after exposing public IP and allowing all traffics, also confirmed they can communicate using socket), they should <strong>be able</strong> to communicate using torch.distributed, right?</li>
</ol>
<p>This question is important since I notice the gloo is also based on the socket, so the answer is “yes” to me. But I want to double-check here.</p>
<p>Update: I use tshark to capture packages between master and worker nodes during the initialization and found they established a connection for one worker port, but this port is different from the one in the error report from the master node.</p>
<p>e.g. master node shows:</p>
<pre data-code-wrap="sh"><code class="lang-nohighlight">RuntimeError: [/opt/conda/conda-bld/pytorch_1634272172048/work/third_party/gloo/gloo/transport/tcp/pair.cc:799] 
connect [13.234.25.113]:48516: Connection refused
</code></pre>
<p>and the tshark capture:</p>
<pre data-code-wrap="sh"><code class="lang-nohighlight">   20 3.422250509 172.31.46.74 → 54.68.21.98  TCP 74 39874 → 23456 [SYN] Seq=0 Win=62727 Len=0 MSS=8961 SACK_PERM=1 TSval=1144394851 TSecr=0 WS=128
   22 3.633177330  54.68.21.98 → 172.31.46.74 TCP 74 23456 → 39874 [SYN, ACK] Seq=0 Ack=1 Win=62643 Len=0 MSS=1460 SACK_PERM=1 TSval=1306173413 TSecr=1144394851 WS=128
   23 3.633196068 172.31.46.74 → 54.68.21.98  TCP 66 39874 → 23456 [ACK] Seq=1 Ack=1 Win=62848 Len=0 TSval=1144395062 TSecr=1306173413
   24 3.633234204 172.31.46.74 → 54.68.21.98  TCP 67 39874 → 23456 [PSH, ACK] Seq=1 Ack=1 Win=62848 Len=1 TSval=1144395062 TSecr=1306173413
   25 3.633240735 172.31.46.74 → 54.68.21.98  TCP 87 39874 → 23456 [PSH, ACK] Seq=2 Ack=1 Win=62848 Len=21 TSval=1144395062 TSecr=1306173413 [TCP segment of a reassembled PDU]
   27 3.844343638  54.68.21.98 → 172.31.46.74 TCP 66 23456 → 39874 [ACK] Seq=1 Ack=2 Win=62720 Len=0 TSval=1306173624 TSecr=1144395062
...
  76 5.014276199 172.31.46.74 → 54.68.21.98  TCP 66 39874 → 23456 [FIN, ACK] Seq=402 Ack=27 Win=62848 Len=0 TSval=1144396443 TSecr=1306174738
</code></pre>
<p><code>54.68.21.98</code> and <code>23456</code> are master address/port. I do not know why the master node reports the port <code>48516</code> goes wrong.  Hope to hear your suggestions. thanks!</p>

username: cbalioglu
cooked: <p><a class="mention" href="/u/huyang719">@HuYang719</a> Note that the master address/port you have specified (i.e. <code>54.68.21.98</code> and <code>23456</code>) are used by the <code>TCPStore</code> that is responsible for establishing a “rendezvous” between workers during process bootstrapping. That socket is not related to Gloo. Once a rendezvous is established, Gloo uses its own socket internally (based on your example looks like it picked the port <code>48516</code> on rank1) for its own communication. You problem still seems to be related to some firewall issue. Are you %100 sure that you allow <strong>all</strong> traffic between the nodes?</p>

username: taehyunzzz
cooked: <p>As <a class="mention" href="/u/cbalioglu">@cbalioglu</a> pointed out for us, the initial master IP and port are used for rendezvous only. Random ports are used for post-rendezvous communications, so you might be having a firewall and port problem.<br>
This could be risky in terms of security, but I solved a similar problem by doing :</p>
<p>node 0 : <code>sudo ufw allow from [node1 IP]</code><br>
node 1 : <code>sudo ufw allow from [node0 IP]</code></p>
<p>I hope there will be some updates for not having to do this… Hope this helps tho <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>