Use of torch.distributed.launch on a distributed multi-node system

dariodematties: <p>Hi, I want to launch 4 processes (two processes per node) on a distributed memory system<br>
Each node in the system has 2 GPUs<br>
So, the layout is the following:<br>
Node 1<br>
rank 0 on GPU:0<br>
rank 1 on GPU:1<br>
Node 2<br>
rank 2 on GPU:0<br>
rank 3 on GPU:1</p>
<p>I am trying to use <a href="https://github.com/pytorch/pytorch/blob/013e6a3d9d0d3ebe8283bf9af0f3b666ba4d96b7/torch/distributed/launch.py#L33" rel="noopener nofollow ugc">this</a> from pytorch documentation</p>
<p>I am using singularity containerization and mpiexec in a script in the following way:</p>
<p>First I do:</p>
<p><code>qsub -n 2 -t 5 -A myproject ./Script.sh</code></p>
<p>which ask for 2 nodes during 5 minutes,</p>
<p>inside the script we have the following command</p>
<p><code>mpiexec -n 4 -f $COBALT_NODEFILE singularity exec --nv -B $mscoco_path $centos_path python3.8 -m torch.distributed.launch --nproc_per_node=2 --nnodes=2 --node_rank=$myrank --master_addr="192.168.1.1" --master_port=1234 $cl_path --b 128 -t -v $mscoco_path</code></p>
<p>How do I get $myrank env variable in order to provide it to --node_rank as stipulated in the <a href="https://github.com/pytorch/pytorch/blob/013e6a3d9d0d3ebe8283bf9af0f3b666ba4d96b7/torch/distributed/launch.py#L33" rel="noopener nofollow ugc">documentation</a>?</p>
<p>Thanks!</p>

mrshenli: <p>Hey <a class="mention" href="/u/dariodematties">@dariodematties</a>, is it possible to know which qsub node (node1 or node2) it is on when running <code>Script.sh</code>? If it is possible to get that, the node rank for the launch script can be derived from node id accordingly?</p>
<p>cc <a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a>, just in case you have seen use cases like this before. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Kiuk_Chung: <p><a class="mention" href="/u/mrshenli">@mrshenli</a> IIUC singularity’s <code>qsub</code> command is invoked from the submitting process. <a class="mention" href="/u/dariodematties">@dariodematties</a> I’m assuming you are using <code>dist.init_process_group(backend="mpi")</code>  inside <code>$mscoco_path</code>?</p>
<p>Since you are already launching with <code>mpiexec</code> there is no need to wrap your script with the distributed launcher. See: <a href="https://github.com/pytorch/pytorch/blob/49f0e5dfeb64a928c6a2368dd5f86573b07d20fb/torch/distributed/distributed_c10d.py#L446" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/49f0e5dfeb64a928c6a2368dd5f86573b07d20fb/torch/distributed/distributed_c10d.py#L446</a></p>
<p>The rank and world size information is provided by mpi runtime.</p>

dariodematties: <p>Thanks <a class="mention" href="/u/mrshenli">@mrshenli</a>, I have tried to get the node id from Script.sh, but I have not been able to do that so far</p>

dariodematties: <p>Hi <a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a>, thank you for your help. I think what you say makes a lot of sense<br>
I tried what you suggest<br>
this is the mpiexec line in Script.sh</p>
<p><code>mpiexec -n 4 -f $COBALT_NODEFILE -ppn 2 singularity exec --nv -B $mscoco_path $centos_path python3.8 $cl_path --b 128 -t -v $mscoco_path</code></p>
<p>I try to launch 4 processes in two nodes (<code>-ppn</code> stands for processes per node)<br>
<code>mscoco_path</code> is a path for the container<br>
<code>centos_path</code> is the path of the container<br>
<code>cl_path</code> is the path to the python script I want to run<br>
what follows are options<br>
I also used <code>dist.init_process_group(backend="mpi")</code> as you suggested<br>
Yet, it is not working</p>
<p>As I can see in the output it launch the container but never launches the python script</p>

Kiuk_Chung: <p><a class="mention" href="/u/dariodematties">@dariodematties</a> In <code>mpiexec -n 4 -f $COBALT_NODEFILE -ppn 2</code>  what does the <code>-n 4</code> and <code>-ppn 2</code> do? My understanding was that <code>mpiexec -n 4 -f $COBALT_NODEFILE</code> is going to invoke 4 procs on each host listed by <code>$COBALT_NODEFILE</code>.</p>

dariodematties: <p>Than you very much for your response <a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a></p>
<p><code>-n 4</code> means 4 processes total and <code>-ppn 2</code> means 2 processes per node</p>
<p>I have been using this extensively in mpi+omp hybrid applications for C and C++ code and this is the behavior</p>
<p><code>mpiexec</code> distributes the 4 processes in the nodes you asked for from <code>qsub</code> using <code>-ppn</code></p>

Kiuk_Chung: <p>apologies for the late reply, circling back on this, were you able to get it working?</p>

dariodematties: <p>All good <a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a></p>
<p>We all are very busy in fact</p>
<p>I have not been able to solve it</p>
<p>Running on a single node I realized that it was not necessary to use <code>mpiexec</code> in front of <code>torch.distributed.launch</code></p>
<p>The correct command to use in the script is the following</p>
<pre><code class="lang-auto">
singularity exec --nv -B $imagenet_path $centos_path python3.8 -m torch.distributed.launch --nproc_per_node=2 $re_path --b 256 -f 2 -v $best_model_path $imagenet_path
</code></pre>
<p>Here I specify 2 processes per node since this node have two GPUs (<code>--nproc_per_node=2</code>)</p>
<p>I launch the scrip using the following line in the command line</p>
<pre><code class="lang-auto">
qsub -n 1 -t 720 -A my-project-name ./run_Active_SimCLR.sh
</code></pre>
<p>Here I am asking for a single node (<code>-n 1</code>) for 12 hours (<code>-t 720</code>)</p>
<p>Remember that I have this running inside a container</p>
<p>As the <a href="https://github.com/pytorch/pytorch/blob/013e6a3d9d0d3ebe8283bf9af0f3b666ba4d96b7/torch/distributed/launch.py#L33" rel="noopener nofollow ugc">documentation</a> points out, when I want to run the same code in two nodes I use the following</p>
<pre><code class="lang-auto">singularity exec --nv -B $mscoco_path $centos_path python3.8 -m torch.distributed.launch --nproc_per_node=2 --nnodes=2 --node_rank=0 --master_addr="192.168.1.1" --master_p    ort=1234 $cl_path --b 128 -t -v $mscoco_path &amp;
singularity exec --nv -B $mscoco_path $centos_path python3.8 -m torch.distributed.launch --nproc_per_node=2 --nnodes=2 --node_rank=1 --master_addr="192.168.1.1" --master_p    ort=1234 $cl_path --b 128 -t -v $mscoco_path
</code></pre>
<p>The new options here that the <a href="https://github.com/pytorch/pytorch/blob/013e6a3d9d0d3ebe8283bf9af0f3b666ba4d96b7/torch/distributed/launch.py#L33" rel="noopener nofollow ugc">documentation</a> specifies are the number of nodes (<code>--nnodes=2</code>) and the node id (<code>--node_rank=0</code>)</p>
<p>And I launch it by</p>
<p><code>qsub -n 2 -t 720 -A my-project-name ./run_Active_SimCLR.sh</code></p>
<p>Where I ask for 2 nodes during 12 hours</p>
<p>Using <code>print</code> statements I realized that the code gets stuck at this line</p>
<p><code>torch.distributed.init_process_group(backend='gloo', init_method='env://')</code></p>
<p>When I check the nodes using <code>qstat</code> they are still listed as running but they are like idle</p>