Questions on underlying port restrictions in nccl/gloo communication

Fei_Liu: <p>Hi,</p>
<p>I have a multi-node task residing on a cluster, and the nodes often failed to do operations like <code>reduce</code> (they hanged there forever). I checked with the network team experts and they told me that it’s because nccl/gloo is using port 0 to be bound with some extra sockets (in addition to the specified <code>MASTER_PORT</code> ), and there is an allowed port range on that cluster so when that port 0 (arbitrary port) falls out of the range, the connection stuck.</p>
<p>Now, to fix this within the pytorch framework: are there any ways to specify this “under-the-hood” port (in addition to the <code>MASTER_PORT</code> ) so that I can make sure that all the port connections are within the allowed range on the cluster?</p>
<p>Thanks!</p>

Fei_Liu: <p>I saw this <code>init_method=“tcp://10.0.0.1:8888”</code> in some other posts (<a href="https://discuss.pytorch.org/t/question-about-torch-distributed-p2p-communication/74560">link</a>). Will something like that work?</p>

wanchaol: <p>init_method would probably work, but can i understand the question more? why would specifying MASTER_PORT not work? I think these two serves similar purposes.</p>

Fei_Liu: <aside class="quote no-group" data-username="wanchaol" data-post="3" data-topic="171573">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wanchaol/48/50343_2.png" class="avatar"> wanchaol:</div>
<blockquote>
<p>ably work, but can i understand the question more? why would specifying MASTER_PORT not work? I think these two serves similar purposes.</p>
</blockquote>
</aside>
<p>I did use MASTER_PORT/MASTER_ADD to control the communication port. What I heard from the network team experts is that there are other communication channels going on by the distributed backend that use port 0 (meaning: arbitrary port) which got blocked randomly. My question is how to control that part?</p>

kumpera: <p>Gloo binds one port per rank to perform p2p communication, it’s not easy to override it since you can have multiple ranks per host.</p>
<p>Unfortunately, we don’t expose in PyTorch any configuration of Gloo’s port binding behavior.</p>
<p>This seems like a problem that could be solved by better configuring your cluster. Is that an option?</p>

Fei_Liu: <p>The network team is working on that but probably not within the near future. Does what you said apply to nccl backend as well? Thanks!</p>

kumpera: <p>NCCL performs p2p communication as well with either sockets or infiniband. <a class="mention" href="/u/kwen2501">@kwen2501</a> should be able to better answer you about it.</p>

kwen2501: <aside class="quote no-group" data-username="Fei_Liu" data-post="1" data-topic="171573">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fei_liu/48/36219_2.png" class="avatar"> Fei_Liu:</div>
<blockquote>
<p>are there any ways to specify this “under-the-hood” port (in addition to the <code>MASTER_PORT</code> )</p>
</blockquote>
</aside>
<p>Hi there,<br>
<code>MASTER_PORT</code> is for PyTorch only, it does not apply to NCCL.<br>
To force NCCL to use a specific port on the master rank, you can use NCCL’s environment variable:<br>
<code>NCCL_COMM_ID=&lt;ip&gt;:&lt;port&gt;</code><br>
For example, <code>NCCL_COMM_ID=192.168.0.1:23456</code></p>
<p>I should note that fixing port via <code>NCCL_COMM_ID</code> has its own side effect:<br>
if you use the <strong>same</strong> port for two consecutive jobs, it could sometimes happen that the OS hasn’t released the port after the first job exits and when the second job starts, hence the second job may complain “Port already in use.”</p>

Fei_Liu: <p>Thank you for the super helpful info! I have a few questions: in the env setting <code>NCCL_COMM_ID=&lt;ip&gt;:&lt;port&gt;</code></p>
<ol>
<li>is the <code>&lt;ip&gt;</code> still the IP of the master (rank 0) machine? (It looks so based on the context, asking to confirm.)</li>
<li>does the <code>&lt;port&gt;</code> need to be<br>
a. the same as the pytorch <code>MASTER_PORT</code> ?<br>
b. an arbitrary one but different from the <code>MASTER_PORT</code> ? Or<br>
c. a completely arbitrary one? (modulo some port range constraints required by the network setup which is why this post/question was brought up in the first place)</li>
<li>I assume the <code>&lt;ip&gt;:&lt;port&gt;</code> still needs to be consistent across all nodes. Is that correct? (Still a fixed choice from the master node.)</li>
</ol>

kwen2501: <p>All good questions <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<ol>
<li>
<p>Yes, usually that’s the case.</p>
</li>
<li>
<p>Please pick a different port than <code>MASTER_PORT </code>; otherwise there would be collision. And yes, it must be a usable port in your network setup.</p>
</li>
<li>
<p>Yes, the same <code>NCCL_COMM_ID</code> must be set on all nodes.</p>
</li>
</ol>

Fei_Liu: <p>Thank you very much!</p>

Fei_Liu: <p>Hi Ke, I’m still running into issues. I tested it with two nodes (dedicated machines that do not have that port restriction), and I kept seeing something like the following:</p>
<pre><code class="lang-auto">... in all_reduce
    work = default_pg.allreduce([tensor], opts)
RuntimeError: NCCL error in: &lt;some hpp file&gt; unhandled system error, NCCL version 2.10.3
ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. It can be also caused by unexpected exit of a remote perr, you can check NCCL warnings for failure reason and see if there is connection closure by a peer.
</code></pre>
<p>My command is like the following:</p>
<pre><code class="lang-auto">NCCL_COMM_ID=&lt;master_addr&gt;:12345 torchrun --node_rank=&lt;0|1&gt; --nnodes=2 --master_addr=&lt;master_addr&gt; --master_port=23456 --nproc_per_node=gpu &lt;some_python_script.py&gt; &lt;script_args...&gt;
</code></pre>
<p>Removing the first <code>NCCL_COMM_ID</code> env then it’ll work fine (though it won’t work on the cluster with the port restrictions).</p>
<p>I also tried to use <code>NCCL_DEBUG=INFO</code>. It says <code>No interface found in the same subnet as remote address &lt;mast_addr&gt;:&lt;12345&gt;</code> and <code> No usable listening interface found</code></p>
<p>Am I missing something?</p>

kumpera: <p>Can you log into those nodes and verify that you can bind to the specified port and connect from the other node to it?</p>
<p>It might be that there’s something in the network configuration that you’re not accounting for.</p>

Fei_Liu: <p>I think the port should be OK. I tried to use the same port in --master_port and it worked fine. Only when I use it in NCCL_COMM_ID (and change --master_port to a different working one) will it stop communicating.</p>

Fei_Liu: <p>I think I get a bit grasp of what’s going on here. I took a deeper look at the NCCL_INFO logs, and found the following:</p>
<p>Suppose I have two nodes (I used variable name to denote all IP/port numbers for convenience):<br>
<strong>rank0</strong> (master), with <strong>ip0</strong><br>
<strong>rank1</strong>, with <strong>ip1</strong><br>
Now, I start the process in both nodes with <strong>–master_port=mpt</strong> (e.g. <strong>mpt=31112</strong>), and <strong>NCCL_COMM_ID=ip0:npt</strong> (where <strong>npt</strong> is a different port number than <strong>mpt</strong>, say <strong>npt=32111</strong>). After the process started, I saw something as follows in the console:</p>
<p>rank0 info log:</p>
<pre><code class="lang-auto">NCCL INFO Bootstrap : Using eth0:ip0&lt;0&gt;
</code></pre>
<p>(note that it’s a port 0 in the <strong>&lt;&gt;</strong> enclosed area)</p>
<p>rank1 info log:</p>
<pre><code class="lang-auto">misc/socket.cc:191 NCCL WARN Net : No interface found in the same subnet as remote address ip0:&lt;npt&gt;
</code></pre>
<p>(note that the port is the selected NCCL port <strong>npt</strong>, not 0).</p>
<p>So somehow, on the <strong>rank0</strong> node, although I have specified to use the NCCL port <strong>npt</strong>, the process seems still using port 0 in the NCCL communication. I think this observation is related to the issue. What do you folks think?</p>

Fei_Liu: <p>I also find something else interesting: if I just set <code>NCCL_COMM_ID=&lt;host's own ip&gt;:&lt;some port&gt;</code>. The process will work ok (communication works), and the info log just says</p>
<pre><code class="lang-auto">NCCL NET/Socket : Using [0]eth0:&lt;host's own ip&gt;&lt;0&gt;
...
NCCL INFO Bootstrap : Using eth0:&lt;host's own ip&gt;&lt;0&gt;
</code></pre>
<p>Same as the case where I don’t specify the <code>NCCL_COMM_ID</code> env var. Overall I didn’t see the NCCL_COMM_ID’s port number appearing anywhere in the info log (using <code>NCCL_DEBUG=INFO</code>), except that when the node1 fails, complaining that it cannot find some interface subnet thing at <code>rank0_ip&lt;nccl_port&gt;</code></p>

kwen2501: <aside class="quote no-group" data-username="Fei_Liu" data-post="15" data-topic="171573">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/fei_liu/48/36219_2.png" class="avatar"> Fei_Liu:</div>
<blockquote>
<pre><code class="lang-auto">misc/socket.cc:191 NCCL WARN Net : No interface found in the same subnet as remote address ip0:&lt;npt&gt;
</code></pre>
</blockquote>
</aside>
<p>This issue may be that host 1 is not in the same subnet as the IP address specified in <code>NCCL_COMM_ID</code>. It is not about the port specified in <code>NCCL_COMM_ID</code>.</p>
<p>For example, if host 0 is in subnet 192.168.1.xx, and host 1 is in subnet 10.0.1.xx, this won’t work.</p>

Fei_Liu: <p>I believe they are in the same subnet. The IPv4 address of the two test nodes only differ in the last 8bit, i.e. the two IPs are <code>a1.a2.a3.b1</code> and <code>a1.a2.a3.b2</code> respectively. This indicates that they are in the same subnet.</p>

sample_y: <p>Maybe you need check net mask use ifconfig, NCCL use net mask check whether two IPs are in same subnet</p>