How to specify MASTER_ADDR and worker ID's for RPC?

ajayp1: <p>I am working with a CPU cluster, where I request a number of nodes to perform model parallelism. I must request jobs and run scripts via slurm. I am unsure how to specify variables such as MASTER_ADDR and MASTER_PORT, and am not sure if I should/how to use the host names of nodes as Worker ID’s (I think I can get these names from the <code>sinfo</code> command).</p>
<p>Also, the worke nodes I’m working with are not connected to the internet, so I’m unsure if I can even specify a MASTER_ADDR/IP address? (I will confirm with the staff)</p>
<p>If anyone has implemented RPC on a cluster with slurm - how did you specific the address/port of the master node and worker id’s of the other nodes? I surely can’t be the first in this situation. I found a <a href="https://discuss.pytorch.org/t/how-to-set-environment-variables-in-torch-nn-parallel-distributeddataparallel/91163">previous post</a> that simply recommended asking the service provider, but it would be nice to know if anyone has handled this issue.</p>

mrshenli: <p>Hey <a class="mention" href="/u/ajayp1">@ajayp1</a>, I am not familiar with slurm, but RPC’s <code>MASTER_ADDR</code> and <code>MASTER_PORT</code> should be similar to the ones used by collective communications and <code>DistributedDataParallel</code> (DDP) in <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.init_process_group" rel="noopener nofollow ugc"><code>init_process_group</code></a>. Have you previously used DDP with slurm?</p>
<p>Found an example here: <a href="https://gist.github.com/TengdaHan/1dd10d335c7ca6f13810fff41e809904" class="inline-onebox" rel="noopener nofollow ugc">Multi-node-training on slurm with PyTorch · GitHub</a></p>

ajayp1: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a>,</p>
<p>Thanks for that example, I was looking all over online for something like that but couldn’t find one.</p>
<p>If anyone else is curious, I used a similar slurm variable to get the worker ID’s and wrote them to a file in my shell script:</p>
<pre><code class="lang-auto">touch /dev/null &gt; $WORKDIR/mynodes.txt
scontrol show hostname $SLURM_JOB_NODELIST &gt; $WORKDIR/mynodes.txt
</code></pre>
<p>Then I simply read from this file in my pytorch script.</p>
<p>To get the rank 0 IP address, I did <code>os.environ['MASTER_ADDR'] = os.system(nodes_list[0] + ' -I awk {print$1}')</code> after reading the rank 0 node ID from my written file. For my specific cluster, the port number didn’t matter, I just picked 8080.</p>
<p>For data parallelism, I’m planning on simply using the distributed library which requires the same MASTER_ADDR, so not expecting a problem there.</p>

mrshenli: <p>Hey <a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a>, is there any recommended way to launch with slurm?</p>

ajayp1: <p>Actually, after reviewing the output file for the RPC implementation, it looks like each node acted as the master, despite the if/else statement to check for rank 0 as per the RPC model parallelism tutorial.</p>
<p>My code is here: <a href="https://github.com/ajayp1/Distributed-Torch-testing" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ajayp1/Distributed-Torch-testing: test scripts for trying out PyTorch's distributed libraries</a></p>
<p>Separate but related issue: when I run the MNIST example from <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html" rel="noopener nofollow ugc">this tutorial</a>, I’m not able to broadcast the MASTER_ADDR and PORT variables to the worker nodes. I managed to at least set the variables on the master node by putting the assignment under <code>if __name__ == "__main__":</code>, but it still doesn’t broadcast to other nodes. Am I supposed to use a certain launcher so that e.g. MPI can communicate between nodes?</p>
<p>My code for this example using just torch.distributed is in the same repo, files have MNIST in the title.</p>

Kiuk_Chung: <p><a class="mention" href="/u/mrshenli">@mrshenli</a> unfortunately we don’t have specific instructions to launch in SLURM. Having said that torchelastic is scheduler agnostic - it uses rendezvous id to “form membership”. So if you follow the instructions in here: <a href="https://pytorch.org/elastic/0.2.1/quickstart.html" class="inline-onebox" rel="noopener nofollow ugc">Quickstart — PyTorch/Elastic master documentation</a> it should work regardless of the scheduler.</p>