Torchrun --rdzv-* and --master* options related questions

username: cangozpi
cooked: <p>Hi I have a couple of questions regarding the torch distributed implementation please bear with me.</p>
<ol>
<li>
<p>why do we have torchrun parameters --rdzv-endpoint and (–master-addr, --master-port) at the same time. I believe --master-* options are for backward compatibility with earlier launch utilities. Am I right ?</p>
</li>
<li>
<p>When I run torchrun as</p>
<pre data-code-wrap="bash"><code class="lang-bash">torchrun --nnodes 1 --nproc_per_node 3  --rdzv-endpoint=localhost:4444 --rdzv-id=7 main.py
</code></pre>
<p>with <em>–rdzv-endpoint</em> only and print the <em>MASTER_PORT</em> environment variable in my code I see that I get <em>MASTER_PORT</em> == 4444 as expected. But when I add the option --rdzv-backend=c10d to the previous command I no longer get <em>MASTER_PORT</em> == 4444 but a random port number (e.g. 33073) at each run different than 4444. Additionally, when I check the ports opened by the python process by running the command:</p>
<pre data-code-wrap="bash"><code class="lang-bash">netstat -tulpn | grep python
</code></pre>
<p>I know see that there are more 2 ports opened (4444, and the random port (e.g. 33073)):<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/a/fa68b1e70b448775345e0321eb13dcd4d457374e.png" data-download-href="https://discuss.pytorch.org/uploads/default/fa68b1e70b448775345e0321eb13dcd4d457374e" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa68b1e70b448775345e0321eb13dcd4d457374e_2_690x35.png" alt="image" data-base62-sha1="zJdFSv9YSsJNGCxuhgCWC6srIce" width="690" height="35" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa68b1e70b448775345e0321eb13dcd4d457374e_2_690x35.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa68b1e70b448775345e0321eb13dcd4d457374e_2_1035x52.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/f/a/fa68b1e70b448775345e0321eb13dcd4d457374e_2_1380x70.png 2x" data-dominant-color="332F30"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1760×90 38.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Why is this so ?</p>
</li>
<li>
<p>What is “static” rendezvous endpoint ? I see it being mentioned as name but couldn’t find an explanation. Even though “static” is the default value for --rdzv-backend, we see the torchrun examples in the documentation pass --rdzv-backend=c10d whenever they are passing --rdzv-backend. Furthermore, in the implementation (<a href="https://github.com/pytorch/pytorch/blob/main/torch/distributed/run.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/distributed/run.py at main · pytorch/pytorch · GitHub</a>), we see that passing the --standalone option does set --rdzv-backend to c10d too:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/a/2aaa8d7fa758aa0064e1dac3c161c8a4581eb5bb.png" data-download-href="https://discuss.pytorch.org/uploads/default/2aaa8d7fa758aa0064e1dac3c161c8a4581eb5bb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2aaa8d7fa758aa0064e1dac3c161c8a4581eb5bb_2_345x204.png" alt="image" data-base62-sha1="65rsBdGce9Hd8fGC71UUhcTNfZh" width="345" height="204" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2aaa8d7fa758aa0064e1dac3c161c8a4581eb5bb_2_345x204.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2aaa8d7fa758aa0064e1dac3c161c8a4581eb5bb_2_517x306.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/a/2aaa8d7fa758aa0064e1dac3c161c8a4581eb5bb_2_690x408.png 2x" data-dominant-color="141920"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">890×528 93.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
So when is it ever used ?</p>
</li>
<li>
<p>In <a href="https://github.com/pytorch/pytorch/blob/main/torch/distributed/run.py" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/distributed/run.py at main · pytorch/pytorch · GitHub</a>, we see that if --rdzv-endpoint option is provided, both the --master-addr and --master-port args are discarded:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec511e0872fb00cf934fde7a1f68b29407356c6c.png" data-download-href="https://discuss.pytorch.org/uploads/default/ec511e0872fb00cf934fde7a1f68b29407356c6c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec511e0872fb00cf934fde7a1f68b29407356c6c_2_345x67.png" alt="image" data-base62-sha1="xIytw1YwywZtYOO1B07f1RPfYrq" width="345" height="67" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec511e0872fb00cf934fde7a1f68b29407356c6c_2_345x67.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec511e0872fb00cf934fde7a1f68b29407356c6c_2_517x100.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec511e0872fb00cf934fde7a1f68b29407356c6c_2_690x134.png 2x" data-dominant-color="1C1E23"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">942×184 46.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
This supports my idea of it being there for backward compatiblity but then the following question of why both the ports specified by passing the options --master-port and --rdzv-endpoint together are all opened by torchrun ?</p>
</li>
<li>
<p>In <a href="https://github.com/pytorch/pytorch/blob/main/torch/distributed/launcher/api.py#L99" class="inline-onebox" rel="noopener nofollow ugc">pytorch/torch/distributed/launcher/api.py at main · pytorch/pytorch · GitHub</a>, we have:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/2/5211e4681489acb2d85b5c0a536a8ca676a30775.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/5211e4681489acb2d85b5c0a536a8ca676a30775" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/5211e4681489acb2d85b5c0a536a8ca676a30775_2_517x285.jpeg" alt="image" data-base62-sha1="bI1u9YD4qn8zjSWisCF4ludmF4V" width="517" height="285" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/5211e4681489acb2d85b5c0a536a8ca676a30775_2_517x285.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/5211e4681489acb2d85b5c0a536a8ca676a30775_2_775x427.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/2/5211e4681489acb2d85b5c0a536a8ca676a30775_2_1034x570.jpeg 2x" data-dominant-color="14171E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1296×716 104 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
as I have mentioned before in my question 3, almost always we pass --rdzv-backend=c10d which is makes code run the following if statement in the above image and return None for master_addr and master_port values:</p>
<pre data-code-wrap="python"><code class="lang-python">if rdzv_parameters.backend != "static":
        return (None, None)
</code></pre>
<p>This discards our --rdzv-endpoint values isn’t this wrong ? I assume it should run as:</p>
<pre data-code-wrap="python"><code class="lang-python">master_addr, master_port = parse_rendezvous_endpoint(endpoint, default_port=-1)
</code></pre>
<p>Am I missing something or is there really an issue here ?</p>
</li>
<li>
<p>Does MASTER_ADDR specify the address that the TCPStore would use to communicate on the master node (rank=0) or the address that the master node would use to communicate with the other processes ? If so then what does --rdzv-endpoint specify ? I though that MASTER_ADDR was used for initializing the TCPStore and --rdzv-endpoint was used for hadling the communication between the different nodes. But in the implementation I see that MASTER_ADDR is discarded for --rdzv-endpoint value (see my question 4) which contradicts my understanding. Could you clarify what these two options are used for ?</p>
</li>
</ol>
<p>Thank you very much for your help, I really appreaciate it.</p>