What's no_sync() exactly do in DDP

cxv0519: <p>I tried DDP no_sync()</p>
<p>well…<br>
I thought no_sync() make not to communicate among ddp group, but it seems not as far as I experimented.</p>
<p>It shows two different operating characteristics in three cases.</p>
<p>Case 1: forward, backward in same no_sync() like below</p>
<pre><code class="lang-auto">        if batch%2==0:
            with ddp_model.no_sync():
                pred = ddp_model(model_in)
                loss = loss_fn(pred, label)
                loss.backward()
        else:        
            pred = ddp_model(model_in)
            loss = loss_fn(pred, label)
            loss.backward()
</code></pre>
<p>It works well.<br>
I mean It accumulates even gradients without synchronization.</p>
<p>Case 2: forward is out of no_sync() and backward is in no_sync() like below</p>
<pre><code class="lang-auto">        pred = ddp_model(model_in)
        if batch%2==0:
            with ddp_model.no_sync():
                loss = loss_fn(pred, label)
                loss.backward()
        else:        
            loss = loss_fn(pred, label)
            loss.backward()
</code></pre>
<p>Well…<br>
In this case, It sync gradients!!</p>
<p>Case3: forward, backward is in no_sync(), but different context like below</p>
<pre><code class="lang-auto">        if batch%2==0:
            with ddp_model.no_sync():
                pred = ddp_model(model_in)
            with ddp_model.no_sync():
                loss = loss_fn(pred, label)
                loss.backward()
        else:        
            pred = ddp_model(model_in)
            loss = loss_fn(pred, label)
            loss.backward()
</code></pre>
<p>In this case, It works like case2; It sync!!</p>
<p>Is there any way to force not to synchronize gradients while decoupling forward and backward?</p>

fduwjj: <aside class="quote no-group" data-username="cxv0519" data-post="1" data-topic="170259">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/cxv0519/48/50136_2.png" class="avatar"> cxv0519:</div>
<blockquote>
<p>out</p>
</blockquote>
</aside>
<p>Thanks for your feedback and question. Looks like this is a DDP specific question, <a class="mention" href="/u/rvarm1">@rvarm1</a> do you mind taking a look?</p>

cxv0519: <p>First of all, thank you so much for your attention.</p>
<p>I could find the relevant annotation in the GitHub.</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py#L1065">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py#L1065" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py#L1065" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py#L1065</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="1055" style="counter-reset: li-counter 1054 ;">
          <li>
          <li>Example::</li>
          <li>
          <li>    &gt;&gt;&gt; # xdoctest: +SKIP("undefined variables")</li>
          <li>    &gt;&gt;&gt; ddp = torch.nn.parallel.DistributedDataParallel(model, pg)</li>
          <li>    &gt;&gt;&gt; with ddp.no_sync():</li>
          <li>    &gt;&gt;&gt;     for input in inputs:</li>
          <li>    &gt;&gt;&gt;         ddp(input).backward()  # no synchronization, accumulate grads</li>
          <li>    &gt;&gt;&gt; ddp(another_input).backward()  # synchronize grads</li>
          <li>
          <li class="selected">.. warning::</li>
          <li>    The forward pass should be included inside the context manager, or</li>
          <li>    else gradients will still be synchronized.</li>
          <li>"""</li>
          <li>old_require_backward_grad_sync = self.require_backward_grad_sync</li>
          <li>self.require_backward_grad_sync = False</li>
          <li>try:</li>
          <li>    yield</li>
          <li>finally:</li>
          <li>    self.require_backward_grad_sync = old_require_backward_grad_sync</li>
          <li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
It says that the context manager should have forward and backward together.</p>
<p>If I follow this context manager decorator function, I think I can get a hint.<br>
If the implementation I want is possible, I will leave an additional reply.</p>