How to correctly launch multi-node training

ankahira: <p><code>dist.get_rank()</code> returns either 0 or 1 even though I launch the training with</p>
<p><code>python -m torch.distributed.launch \    --nproc_per_node=4 \ 	 --nnodes=2 \ 	 --node_rank=0 \ 	 --master_addr="$MASTER_ADDR" \ 	 --master_port="$MASTER_PORT" \    train_dist.py &amp; </code></p>
<p>I am using 2 nodes each with 4 GPUs.</p>

mrshenli: <p>Hey <a class="mention" href="/u/ankahira">@ankahira</a></p>
<p>Two questions:</p>
<ol>
<li>What parameters did you pass to <code>init_process_group</code> invocation in <code>train_dist.py</code>?</li>
<li>Can you check if <code>RANK</code> and <code>WORLD_SZIE</code> are set properly for each process?</li>
</ol>
<p>Sudarshan wrote a <a href="https://github.com/pytorch/examples/pull/743" rel="nofollow noopener">great example</a> of how to use launcher.py, which might be helpful to you.</p>

ankahira: <p>Actually I am abit confused about this. I understand that I should set <code>WORLD-SIZE</code> as Number of nodes i.e 2. I am not sure what I should set as <code>RANK</code> but I set it as 0 and 1 for the two nodes. For the <code>init_process</code>_group` I pass each of the GPUs as in 0, 1, 2 ,3.  Something like this.</p>
<p><code>def init_processes(rank, size, fn, backend='nccl'):     dist.init_process_group(backend, rank=rank, world_size=size)     fn(rank, size)</code></p>
<p><code>if __name__ == "__main__": </code><br>
<code> size = 4     processes = []     for rank in range(size):         p = Process(target=init_processes, args=(rank, size, run))         p.start()         processes.append(p)</code></p>
<pre><code>for p in processes:
    p.join()`</code></pre>

mrshenli: <aside class="quote no-group" data-username="ankahira" data-post="3" data-topic="74482">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ankahira/48/21299_2.png" class="avatar"> ankahira:</div>
<blockquote>
<p>I should set <code>WORLD-SIZE</code> as Number of nodes i.e 2. I am not sure</p>
</blockquote>
</aside>
<p>If you have 8 processes (4 processes per node with 2 node), world_size should be 8 for <code>init_process_group</code>. But you don’t need to set that, as the launcher script will set the env vars for you properly.</p>
<blockquote>
<p>I am not sure what I should set as  <code>RANK</code>  but I set it as 0 and 1 for the two nodes</p>
</blockquote>
<p>There are two ranks here:</p>
<ul>
<li>node rank: this is what you provide for <code>--node_rank</code> to the launcher script, and it is correct to set it to 0 and 1 for the two nodes.</li>
<li>process rank: this rank should be <code>--node_rank</code> X <code>--nproc_per_node</code> + local GPU id, which should be 0~3 for the four processes in the first node, and 4~7 for the four processes in the second node. But you don’t need to set this for <code>init_process_group</code> either, as the launcher script should have set the env var for you.</li>
</ul>
<p>With the params you provided to the launcher script, the following should be sufficient to init the process group.</p>
<pre><code class="lang-auto">dist.init_process_group(backend)
</code></pre>
<p>If you also need the local rank for DDP, you will need to parse it from the arg, and then pass it to the DDP constructor. Sth like:</p>
<pre><code class="lang-auto">    model = torch.nn.parallel.DistributedDataParallel(model,
                                                      device_ids=[arg.local_rank],
                                                      output_device=arg.local_rank)
</code></pre>
<p>Check out the <a href="https://github.com/pytorch/pytorch/blob/f326045b3757236aabe367dfca1894be14ce31ef/torch/distributed/launch.py#L116-L118" rel="noopener nofollow ugc">readme</a> in the launcher script.</p>

ankahira: <p>Thank you very much. That is very clear now.</p>

Pooria_Taheri: <p><a class="mention" href="/u/ankahira">@ankahira</a> Can you please share your train_dist.py please? I still have problem with using the luncher!</p>