Distributed Data Parallel no_sync

lkp411: <p>Are forward and backward still synchronization points in DDP even if they are inside a no_sync() context? My understanding is that no_sync prevents gradient averaging, but I was wondering if it disables syncing completely.</p>

mrshenli: <p>Hey <a class="mention" href="/u/lkp411">@lkp411</a></p>
<p><code>no_sync()</code> disables the following code, which should completely disable communications across processes (unless you are using <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel.join" rel="noopener nofollow ugc">DDP.join()</a>).</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/b1907f5ebcaeb7405aa3d9d8025a7a5eeb0ee590/torch/nn/parallel/distributed.py#L710-L720" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/b1907f5ebcaeb7405aa3d9d8025a7a5eeb0ee590/torch/nn/parallel/distributed.py#L710-L720" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/b1907f5ebcaeb7405aa3d9d8025a7a5eeb0ee590/torch/nn/parallel/distributed.py#L710-L720</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="710" style="counter-reset: li-counter 709 ;">
<li>if torch.is_grad_enabled() and self.require_backward_grad_sync:</li><li>    self.require_forward_param_sync = True</li><li>    # We'll return the output object verbatim since it is a freeform</li><li>    # object. We need to find any tensors in this object, though,</li><li>    # because we need to figure out which parameters were used during</li><li>    # this forward pass, to ensure we short circuit reduction for any</li><li>    # unused parameters. Only if `find_unused_parameters` is set.</li><li>    if self.find_unused_parameters:</li><li>        self.reducer.prepare_for_backward(list(_find_tensors(output)))</li><li>    else:</li><li>        self.reducer.prepare_for_backward([])</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


lkp411: <p>Hey <a class="mention" href="/u/mrshenli">@mrshenli</a></p>
<p>Thanks so much for you reply. What are the other sync points in DDP? I remember reading somewhere here that copying data from the gpu to the cpu forces a sync. Is this true? And does it still happen in a no_sync context?</p>
<p>Once again, thanks for your time.</p>

ptrblck: <aside class="quote no-group" data-username="lkp411" data-post="3" data-topic="110555">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/l/90ced4/48.png" class="avatar"> lkp411:</div>
<blockquote>
<p>I remember reading somewhere here that copying data from the gpu to the cpu forces a sync. Is this true? And does it still happen in a no_sync context?</p>
</blockquote>
</aside>
<p>Yes, this is true, since you would need the calculated tensor in order to push it to the CPU (thus the CPU has to synchronize on the GPU workload).<br>
This should also be the case in a DDP setup with <code>no_sync()</code>.</p>

mrshenli: <p>Hey <a class="mention" href="/u/lkp411">@lkp411</a>, in the context of DDP, there are two different types of synchronizations, intra-process (CUDA stream) and inter-process (collective comm). The gradient averaging (<code>AllReduce</code>) is inter-process sync and the CPU-to-GPU copy is a intra-process sync. Which ones are you referring to in the following question?</p>
<blockquote>
<p>What are the other sync points in DDP?</p>
</blockquote>

lkp411: <p>I was referring to interprocess syncs. Are there any other sync points besides the ddp constructor, forward, backward and collective communication calls? And does calling torch.cuda.synchronize(rank) in a specific process after issuing an async collective communication call (for example an all_reduce) block till the result of the collective communication call is available?</p>
<p>Also since we’re on the topic, are there plans to add sparse all_reduce capabilities to the NCCL backend?</p>