Does DistributedDataParallel calculate the average gradient across each GPU or each node?

mesllo: <p>Whatever article, GitHub issue, or Pytorch/StackOverflow question I read, I still can’t seem to get a clear-cut answer on this.</p>
<p>Suppose I have the following scenario:</p>
<p><strong>Scenario:</strong> 1 node, 4 GPUs, batch size M</p>
<p>According to this <a href="https://pytorch.org/docs/stable/notes/ddp.html" rel="noopener nofollow ugc">documentation</a>, DDP will create 4 processes each having their own dataloader and model. On most official Pytorch implementations of DDP (eg. Imagenet models), common practice is to <strong>divide the batch size by the number of GPUs per node</strong>. This means that each dataloader (each process) will take batches of size <strong>M/4</strong>. Further on in this article, it explains that the following:</p>
<p><em>When gradients in one bucket are all ready, the <code>Reducer</code> kicks off an asynchronous <code>allreduce</code> on that bucket to calculate mean of gradients across all processes. When all buckets are ready, the <code>Reducer</code> will block waiting for all <code>allreduce</code> operations to finish. When this is done, averaged gradients are written to the <code>param.grad</code> field of all parameters.</em></p>
<p><strong>Question 1:</strong> What is a bucket here? Is it a collection of training examples (eg. 1 per process or 1 per GPU?)<br>
<strong>Question 2:</strong> Since we take the average gradient across processes that have a batch size of <strong>M/4</strong> isn’t this incorrect since we actually want the average gradient of batch size of <strong>M</strong>?<br>
<strong>Question 3:</strong> In a multi-node setting, to be consistent with the above explanation, shouldn’t we divide by the total number of GPUs rather than by the GPUs per node? Eg. for 2 nodes and 4 GPUs, shouldn’t we divide <strong>M/8</strong>?<br>
<strong>Question 4:</strong> In general for N nodes having M GPUs each, how should we divide the batch size across the nodes and GPUs <strong>M</strong> if we want our training to <strong>reflect the results of mini-batch GD with batch size M</strong>?</p>

rvarm1: <ol>
<li>A bucket is DDP’s notion of a collection of parameter gradients, so a bucket can essentially contain 1 or more parameter grads.</li>
<li>Yeah, if loss is summed across instances, gradient will be 4 times smaller. Please see the note here: <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/parallel/distributed.py#L305" class="inline-onebox" rel="noopener nofollow ugc">pytorch/distributed.py at master · pytorch/pytorch · GitHub</a>
</li>
<li>We divide by the number of training processes, so if we have 2 nodes and 4 GPUs per node, we indeed divide by 8.</li>
<li>In this case, you have M * N workers. Given a local batch size B, you can have a batch B /(M * N) on each node, if loss is summed instead of averaged, you may have to scale the gradients by the world size N.</li>
</ol>

mesllo: <blockquote>
<p>In this case, you have M * N workers. Given a local batch size B, you can have a batch B /(M * N) on each node, if loss is summed instead of averaged, you may have to scale the gradients by the world size N.</p>
</blockquote>
<p>Why each node? If we have batch size B, M GPUs, N nodes, shouldn’t we divide like B/(M<em>N) on <strong>each gpu</strong>? So each GPU out of the M</em>N GPUs will take in batch size B/(M*N)</p>