SLURM srun vs torchrun: Difference numbers of spawn processes

tallgeese: <p>Hi,</p>
<p>For a situation of two computing nodes having 4 GPUs each, I could not reconcile the difference in number of spawn processes resulting from a directly launch  with <code>srun</code> and delegated one with <code>torchrun</code></p>
<ol>
<li>Direct launch with SLURM’s <code>srun</code></li>
</ol>
<pre><code class="lang-auto">import torch.distributed as dist

WORLD_SIZE = int(os.environ['SLURM_NTASKS'])
WORLD_RANK = int(os.environ['SLURM_PROCID'])
LOCAL_RANK =int(os.environ['SLURM_LOCALID'])
  
dist.init_process_group('nccl', rank=WORLD_RANK, world_size=WORLD_SIZE)

device = torch.device("cuda:{}".format(LOCAL_RANK))
</code></pre>
<p>Here, 2 x 4 = 8 processes are  required in total, i.e.</p>
<pre><code class="lang-auto">$ srun -n 8 python test.py
</code></pre>
<ol start="2">
<li>Delegated launch via <code>torchrun</code></li>
</ol>
<p>According to manual, <code>WORLD_SIZE</code>, <code>RANK</code>, and <code>LOCAL_RANK</code> are automatically populated by <code>torchrun</code>.</p>
<pre><code class="lang-auto">import torch.distributed as dist

dist.init_process_group('nccl', init_method="env://")

device = torch.device("cuda:{}".format(os.environ['LOCAL_RANK']))
</code></pre>
<p>To launch <code>torchrun</code> via SLURM scheduler, I need to use a wrapper to asign <code>--node-rank</code> correctly, i.e.</p>
<pre><code class="lang-auto">$ srun -n 2 wrapper.sh 
</code></pre>
<p>With the content of <code>wrapper.sh</code> as follow:</p>
<pre><code class="lang-auto">torchrun \
    --nnodes=$SLURM_NNODES \
    --node_rank=$SLURM_NODEID \
    --nproc_per_node=$SLURM_GPUS_ON_NODE \
     test.py
</code></pre>
<p>Thus:</p>
<ul>
<li>(1) spawn 8 tasks / 8 cores → 1 task per core</li>
<li>(2) spawn 8 tasks / 8 cores → 4 tasks per core</li>
</ul>
<p>The outputs are same, yet the allocated resources are difference.<br>
I am somewhat confused regarding the performance implication.<br>
(1) is probably better due to the absence of oversubscription.<br>
In other words, there is no point of using  <code>torchlaunch</code> with fixed resource allocation via SLURM.</p>
<p>I appreciate your insights on this matter.<br>
Thanks.</p>

biubiu: <p>did you figure out the reason? I ran into a similar issue: 2 nodes and 4GPUs per node. I started torchrun on each node, everything works fine (when I check CPU usage of each python process, they are around 100%~200%). But if I use slurm (18.08) with srun, the 4 python processes have a net CPU usage 100% (about 25% each). Here is the submitting script.</p>
<p>#!/bin/bash<br>
<span class="hashtag-raw">#SBATCH</span> --job-name=“test”<br>
<span class="hashtag-raw">#SBATCH</span> -p doppler<br>
<span class="hashtag-raw">#SBATCH</span> --cpus-per-task=40<br>
<span class="hashtag-raw">#SBATCH</span> --gpus-per-task=4<br>
<span class="hashtag-raw">#SBATCH</span> --nodes=2<br>
<span class="hashtag-raw">#SBATCH</span> --ntasks=2</p>
<h6><a name="p-464143-set-the-env-for-torch-ddp-1" class="anchor" href="#p-464143-set-the-env-for-torch-ddp-1"></a>Set the Env. for torch DDP</h6>
<p>export MASTER_PORT=$(expr 10000 + $(echo -n $SLURM_JOBID | tail -c 4))<br>
export WORLD_SIZE=$(($SLURM_NNODES * $SLURM_GPUS_PER_TASK))<br>
master_addr=$(scontrol show hostnames “$SLURM_JOB_NODELIST” | head -n 1)<br>
export MASTER_ADDR=$master_addr<br>
export OMP_NUM_THREADS=4</p>
<p>srun --chdir=/mylocaldir --export=ALL <br>
torchrun <br>
–nnodes=$SLURM_JOB_NUM_NODES <br>
–nproc_per_node=4 <br>
–rdzv_id=$SLURM_JOB_ID <br>
–rdzv_backend=c10d <br>
–rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT <br>
main.py</p>
<p>Any suggestions will be appreciated!</p>

tallgeese: <p>Hi <a class="mention" href="/u/biubiu">@biubiu</a></p>
<p>Sorry for belated reply.<br>
It has been a long time since the original question.</p>
<p>My understanding is that I cannot launch torchrun directly via srun, i.e.</p>
<pre><code class="lang-auto">srun torchrun --
</code></pre>
<p>This is because the <code>$SLURM_NODEID</code> is not propagated correctly to <code>torchrun</code>.<br>
I had to use <code>wrapper.sh</code> where torchrun will inherit <code>$SLURM_NODEID</code> from the parent <code>wrapper.sh</code></p>
<p>Could you try launching through a wrapper ?</p>
<p>Since I don’t need to worry about reliability in case one of the node is down, I opts to explicitly initialize DDP inside the training script nowadays.</p>
<p>Hope it help.</p>