How average gradients compute when DDP model run in different speed?

LW-Ricarido: <p>Hi, I have a question about DDP computing average. I read in <a href="https://pytorch.org/docs/stable/notes/ddp.html" rel="noopener nofollow ugc">DDP backward pass</a> that says when all buckets are ready, local <code>Reducer</code> will block waiting for all <code>allreduce</code> to opertions to finish. But what will happen when several GPU run in different speed? For example, when two GPUs(e.g. <code>cuda:0</code> and <code>cuda:1</code>) run 1.5x faster than other GPUs(codes and processing are the same), <code>cuda:0</code> and <code>cuda:1</code> will produce more gradients. Will they save these gradients in the bucket and wait for other GPUs to get ready, or they just abandon these gradients and reducer gradients that are ready in all GPUs?</p>
<p>Thanks a lot.</p>

seungjun: <p>The faster GPU processes will wait for other GPUs to finish their backward computation.<br>
By default, DDP synchronizes gradients and parameters and then performs the next forward computation.<br>
The differing speed case is quite common in practice.</p>
<p>You can take a look at the forward function of DDP.<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py#L675" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py#L675" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py#L675</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="665" style="counter-reset: li-counter 664 ;">
<li># Calling _rebuild_buckets before forward compuation,</li><li># It may allocate new buckets before deallocating old buckets</li><li># inside _rebuild_buckets. To save peak memory usage,</li><li># call _rebuild_buckets before the peak memory usage increases</li><li># during forward computation.</li><li># This should be called only once during whole training period.</li><li>if self.reducer._rebuild_buckets():</li><li>    logging.info("Reducer buckets have been rebuilt in this iteration.")</li><li></li><li>if self.require_forward_param_sync:</li><li class="selected">    self._sync_params()</li><li></li><li>if self.ddp_uneven_inputs_config.ddp_join_enabled:</li><li>    # Notify joined ranks whether they should sync in backwards pass or not.</li><li>    self._check_global_requires_backward_grad_sync(is_joined_rank=False)</li><li></li><li>if self.device_ids:</li><li>    if len(self.device_ids) == 1:</li><li>        inputs, kwargs = self.to_kwargs(inputs, kwargs, self.device_ids[0])</li><li>        output = self.module(*inputs[0], **kwargs[0])</li><li>    else:</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


osalpekar: <p>Just to add some more insight to this, we have a <code>bucket_cap_mb</code> argument in the DDP constructor. This defines the size of a gradient bucket in megabytes. During the backward pass, each rank fills the bucket with gradients and then kicks off the allreduce collective. Faster ranks will kick off the allreduce collective earlier than the slower ranks, so they will just block until the slower ranks kick off the collective. No gradients are abandoned in this process. You can tune the <code>bucket_cap_mb</code> as desired, and this will trigger allreduce more frequently for smaller buckets and less frequently for larger buckets.</p>
<p>If the performance difference is too great, you can explore syncing gradients less frequently (every <code>n</code> batches instead of every batch) using the <code>model.no_sync()</code> context manager or using multiple process groups (using the <code>new_group</code> API). If you find DDP training getting stuck due to excessively long hang-times due to these blocked collectives, you may look into using torchelastic and some mechanism to timeout hanging collectives (such as <code>NCCL_ASYNC_ERROR_HANDLING</code> or <code>NCCL_BLOCKING_WAIT</code> - <a href="https://pytorch.org/docs/stable/distributed.html?highlight=init_process_group#torch.distributed.init_process_group" rel="noopener nofollow ugc">docs here</a>)</p>