Torch.dist.distributedparallel vs horovod

bkuriach: <p>What is the key difference between <strong>torch.dist.distributedparallel</strong> and <strong>horovod</strong>?<br>
If my understanding is correct, torch.dist.distributedparallel work on single node with one or more GPUs (it does not distribute workloads across GPUs across more than one node) whereas horovod can work with multi-node multi-gpu.</p>
<p>If my understanding is not correct, kindly explain when to use horovod and when to use torch.dist.distributedparallel?</p>
<p>Kindly share your thoughts? Thank you very much in advance!!</p>

ptrblck: <p>As given in the <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html">DDP docs</a>, <code>DistributedDataParallel</code> is able to use multiple machines:</p>
<blockquote>
<p><a href="https://pytorch.org/docs/stable/nn.html#torch.nn.parallel.DistributedDataParallel">DistributedDataParallel</a> (DDP) implements data parallelism at the module level which can run across multiple machines. Applications using DDP should spawn multiple processes and create a single DDP instance per process.</p>
</blockquote>
<p>I’m not familiar with <code>horovod</code> and don’t know what the advantages might be.</p>
<p>PS: please don’t tag specific users, as it might discourage others to post better answers <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>

bkuriach: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thank you very much for your response!!</p>

mrshenli: <p>One difference between PyTorch DDP is Horovod+PyTorch is that, DDP overlaps backward computation with communication. In contrast, according to the following example, Horovod synchronizes models in the optimizer <code>step()</code>, which won’t be able to overlap with backward computations. So, in theory, DDP should be faster.</p>
<p><a href="https://horovod.readthedocs.io/en/stable/pytorch.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://horovod.readthedocs.io/en/stable/pytorch.html</a></p>

ducviet00: <p>I don’t think so. Horovod is able to create async communication functions for parameter.grad’s hook to synchronize gradients. That gives handles of async functions, in optimizer.step(), they synchronize them so that overlap backward.</p>

irasit: <p>Here is some comparison from horovod:</p><aside class="onebox githubissue" data-onebox-src="https://github.com/horovod/horovod/issues/1973">
  <header class="source">

      <a href="https://github.com/horovod/horovod/issues/1973" target="_blank" rel="noopener nofollow ugc">github.com/horovod/horovod</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/horovod/horovod/issues/1973" target="_blank" rel="noopener nofollow ugc">Comparison with pytorch self distributed training</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-05-21" data-time="12:45:36" data-timezone="UTC">12:45PM - 21 May 20 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2020-05-30" data-time="08:20:49" data-timezone="UTC">08:20AM - 30 May 20 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/songtaoshi" target="_blank" rel="noopener nofollow ugc">
          <img alt="songtaoshi" src="https://avatars.githubusercontent.com/u/20240391?v=4" class="onebox-avatar-inline" width="20" height="20">
          songtaoshi
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          question
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">Hi, I am a little confused about the benchmark comparison with pytorch self dist<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">ributed training.  
Because recently I am surveying the best practice for distributed training and I see horovod, but I do not know what the superior in effect and speed with pytorch self distributed training. And I know your horovod is very concise for using.</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


whatdhack: <p>is there a DDP control plane like in Horovod ?  Where do I find  end to end architectural documentation of DDP ?</p>