How to map processes to GPU in DDP and how to launch the DDP cluster?

username: Olivier-CR
cooked: <p>Hi,</p>
<p>I have 2 EC2 machines with 4 GPUs each. That makes 8 GPUs in total.</p>
<p>I want to train a PyTorch DeepLab in data-parallel over those 8 cards. What should I do:</p>
<p>A. Launch a DDP training with 2 scripts/processes (1 per node), each doing torch.nn.DataParallel to data-parallel within node on the 4 cards</p>
<p>B. Launch a DDP training with 8 scripts/processes (1 per GPU), each executing pure DDP + PyTorch code and using only 1 GPU (leaving DDP doing the allreduces).</p>
<p>In both options: how to launch the processes with torchrun/launch.py/torch.distributed: once for the whole cluster, from some remote client? Once per node? Once per GPU?..</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/olivier-cr">@Olivier-CR</a></p>
<blockquote>
<p>A. Launch a DDP training with 2 scripts/processes (1 per node), each doing torch.nn.DataParallel to data-parallel within node on the 4 cards</p>
</blockquote>
<p><code>DataParallel</code> is single-machine multi-GPU. It won’t work in the multi-machine scenario. <code>DistributedDataParallel</code> is the appropriate feature to use.</p>
<blockquote>
<p>B. Launch a DDP training with 8 scripts/processes (1 per GPU), each executing pure DDP + PyTorch code and using only 1 GPU (leaving DDP doing the allreduces).</p>
</blockquote>
<p>Yep, this should work. One caveat is that you need to make sure each DDP process exclusively operate on a dedicated GPU. You can do this by either setting the <code>CUDA_VISIBLE_DEVICES</code> to different GPU for different processes, or use the <code>local_rank</code> within each process. See: <a href="https://pytorch.org/docs/master/distributed.html#launch-utility" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch master documentation</a></p>
<blockquote>
<p>In both options: how to launch the processes with torchrun/launch.py/torch.distributed: once for the whole cluster, from some remote client? Once per node? Once per GPU?..</p>
</blockquote>
<p>If you are using torch <code>run.py</code>/<code>launch.py</code>, you just need to do it once for each machine. If you directly call your user script without run/launch, you will need to do that once for each GPU.</p>
<p>I do recommend using <a href="https://pytorch.org/docs/stable/distributed.elastic.html" rel="noopener nofollow ugc"><code>TorchElastic</code></a> to launch jobs, as that will also provide failure recoveries.</p>
<p>Hey <a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a> what’s the best tutorial to get started with <code>run</code>/<code>launch</code>? Thanks!</p>

username: Kiuk_Chung
cooked: <p><a class="mention" href="/u/mrshenli">@mrshenli</a> , <a class="mention" href="/u/olivier-cr">@Olivier-CR</a> the <code>torch.distributed.run</code> docs isn’t a tutorial but a great place to start: <a href="https://pytorch.org/docs/stable/elastic/run.html" class="inline-onebox" rel="noopener nofollow ugc">torchrun (Elastic Launch) — PyTorch 1.10.0 documentation</a></p>
<p>For a more out-of-the-box experience, TorchX is where we are trying to make distributed job launching much easier: <a href="https://pytorch.org/torchx/main/components/distributed.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed — PyTorch/TorchX main documentation</a></p>

username: Olivier-CR
cooked: <p>wow what is that torchX thing? some other new library again <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"> ?! I though anything distributed would go to TorchElastic / Torchrun now? I’m a bit confused. Or is it just a thick client to launch distributed jobs?</p>

username: Kiuk_Chung
cooked: <p>torchrun will launch LOCAL processes for you. To run a distributed job its still on you to run <code>torchrun</code> on each of the nodes. TorchX has builtins to launch the job for you, and in doing so sets sensible defaults so that you don’t have to manually set configurations like  <code>--rdzv_backend</code>, <code>--rdzv_id</code> etc.</p>
<p>Try:</p>
<pre><code class="lang-auto">$ pip install torchx-nightly
$ torchx run -s local_cwd dist.ddp -j 1x4 --script YOUR_SCRIPT.py &lt;args to script&gt;
</code></pre>
<p>Where <code>-j</code> is of the form <code>{nnodes}x{nproc_per_node}</code> so if you wanted to simulate a 2 node (each node running 4 procs per node) you’d set <code>-j 2x4</code>.</p>
<p>No need to worry about the different rendezvous settings now. Once you are ready to submit to a remote cluster (assuming you’ve setup kubernetes or slurm) then you’d run</p>
<pre><code class="lang-auto">$ torchx run -s kubernetes dist.ddp -j 1x4 --image YOUR_DOCKER_IMG --script SCRIPT.py &lt;args to script&gt;
</code></pre>

username: Olivier-CR
cooked: <p><a class="mention" href="/u/kiuk_chung">@Kiuk_Chung</a> what if you want to run via torchX on a plain EC2 cluster? still have to install etcd servers? <em>“(needs you to start a local etcd server on port 2379! and have a <code>python-etcd</code> library installed)”</em> (<a href="https://pytorch.org/torchx/main/components/distributed.html" rel="noopener nofollow ugc">from here</a>)</p>