Decoding the different methods for multi-NODE distributed training

mohotmoz: <p>Hello everyone,</p>
<p>Thank you in advance for your time!</p>
<p>I am trying to wrap my head around all of the different distributed training libraries out there, specifically for multi node training. I understand that Pytorch DDP can operate in a multi-node fashion.</p>
<p>Sorry for the vague-ness here… I think I am just having trouble understanding the difference between:</p>
<ol>
<li>torch.DistributedDataParallel</li>
<li>deepspeed</li>
<li>horovod</li>
<li>fairscale</li>
<li>nvidia apex (apex.parallel.DistributedDataParallel)</li>
<li>ray</li>
<li>raySGD</li>
<li>MPI (totally get this for big CPU cluster jobs from like 20 years ago… how does it fit in here?)</li>
</ol>
<p>The closest I have come to understanding how everything connects is from this blog post:<br>
<a href="https://medium.com/distributed-computing-with-ray/faster-and-cheaper-pytorch-with-raysgd-a5a44d4fd220" class="onebox" target="_blank" rel="noopener nofollow ugc">https://medium.com/distributed-computing-with-ray/faster-and-cheaper-pytorch-with-raysgd-a5a44d4fd220</a></p>
<p>They say:</p>
<blockquote>
<p>You can use one of the integrated tools for doing distributed training like <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="noopener nofollow ugc">Torch Distributed Data Parallel</a> or <a href="https://www.tensorflow.org/guide/distributed_training" rel="noopener nofollow ugc">tf.Distributed</a>. While these are “integrated”, they are certainly not a walk in the park to use.</p>
<p><a href="https://pytorch.org/tutorials/beginner/aws_distributed_training_tutorial.html" rel="noopener nofollow ugc">Torch’s AWS tutorial</a> demonstrates the many setup steps you’re going to have to follow to simply get the cluster running, and Tensorflow 2.0 has <a href="https://github.com/tensorflow/tensorflow/issues?q=is%3Aissue+is%3Aopen+label%3Acomp%3Adist-strat" rel="noopener nofollow ugc">a bunch of issues</a>.</p>
<p>Maybe you might look at something like <a href="https://github.com/horovod/horovod" rel="noopener nofollow ugc">Horovod</a>, but Horovod is going to require you to fight against antiquated frameworks like MPI and wait a long time for compilation when you launch.</p>
</blockquote>
<p>I also watched this video which explains some of the differences:</p><div class="youtube-onebox lazy-video-container" data-video-id="3XUG7cjte2U" data-video-title="Invited Talk: PyTorch Distributed (DDP, RPC) - By Facebook Research Scientist Shen Li" data-video-start-time="" data-provider-name="youtube">
  <a href="https://www.youtube.com/watch?v=3XUG7cjte2U" target="_blank" rel="noopener nofollow ugc">
    <img class="youtube-thumbnail" src="https://img.youtube.com/vi/3XUG7cjte2U/maxresdefault.jpg" title="Invited Talk: PyTorch Distributed (DDP, RPC) - By Facebook Research Scientist Shen Li" width="690" height="388">
  </a>
</div>

<p>I was wondering if someone who knows a lot more about this than I do could clear up how <span class="hashtag-raw">#1-</span><span class="hashtag-raw">#8</span> relate to each other! And bonus points if you have a good rule of thumb for when to use which one! My/our applications are pretty straightforward - (NLP stuff, image–&gt;text stuff)… just have a ton of data and trying to speed things up by going multi-node… then ran into <span class="hashtag-raw">#1-</span><span class="hashtag-raw">#8</span> and got confused and figured maybe clearing it up would help others as well!</p>
<p>Thanks again everyone.</p>

ptrblck: <p>I can’t comment on all libs, but <span class="hashtag">#1</span> vs. <span class="hashtag">#5:</span><br>
<code>apex.DDP</code> is deprecated in favor of the native PyTorch <code>DistributedDataParallel</code>. Similar to the mixed-precision training (<code>amp)</code> <code>apex</code> provided <code>DDP</code> while there shouldn’t be any advantage of using it anymore and you should thus switch the the native <code>DDP</code> implementation.</p>