Node -to-node communication traffic in Distributed Data Parallel & RPC Model Parallel

username: ZiyiZhu
cooked: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a>,</p>
<p>I wonder if I can ask some questions on the implementation details of the communication traffic for both DDP data-parallel and RPC model-parallel using multiple machines.</p>
<p>In the documentation, <a href="https://pytorch.org/docs/stable/notes/ddp.html?highlight=ddp" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/ddp.html?highlight=ddp</a>, if I understand correctly the .backforward() will work on the local gradients and the DDP will help to synchronize the local gradients with the gradients in other nodes (not sure, perhaps ring-all reduce?). For the SGD optimizer as an example, the local gradients are first averaged over the batch size, and then the locally averaged gradients will be sent to other nodes in buckets by <strong>DDP</strong>. The traffic size would be roughly the size of the NN’s gradient size of a batch size of 1. Do I understand this correctly? Or some optimizers or networks do need to send some size (gradients * batch size)?</p>
<p><a href="https://pytorch.org/docs/stable/rpc.html" rel="nofollow noopener">https://pytorch.org/docs/stable/rpc.html</a> For the RPC, the <strong>Distributed Autograd</strong> will take care of the communication in the backward direction at the RPC boundaries of each machine. So the communication happens when the user code calls .to_here for the <strong>RREF</strong> and the <strong>Distributed Autograd</strong> passes the gradients in backward? The traffic does relate to the batch size * the size of feature maps at the boundaries?</p>
<p>Please let me know if I understand the mechanisms of the communication behind <strong>DDP</strong> and <strong>RPC</strong> correctly. Thank you very much for your time!</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/ziyizhu">@ZiyiZhu</a></p>
<blockquote>
<p>the .backforward() will work on the local gradients and the DDP will help to synchronize the local gradients with the gradients in other nodes (not sure, perhaps ring-all reduce?).</p>
</blockquote>
<p>Yes, this is correct. By default Gloo uses ring allreduce IIUC. For NCCL, you can configure <a href="https://docs.nvidia.com/deeplearning/sdk/nccl-developer-guide/docs/env.html#nccl-tree-threshold" rel="noopener nofollow ugc"><code>NCCL_TREE_THRESHOLD</code></a> to tell NCCL when to enable tree allreduce. (I haven’t tested this myself yet.)</p>
<blockquote>
<p>For the SGD optimizer as an example, the local gradients are first averaged over the batch size, and then the locally averaged gradients will be sent to other nodes in buckets by  <strong>DDP</strong> . The traffic size would be roughly the size of the NN’s gradient size of a batch size of 1. Do I understand this correctly?</p>
</blockquote>
<p>Yes, one minor note is that the batch size does not affect the amount of data sent over network, because <code>AccumulateGrad</code> will accumulate gradients from different samples into the same <code>param.grad</code> tensor. So the data sent over wire is always <code>param.grad</code> for all params in <code>model.parameters()</code>. And yes, gradients are organized into buckets to speed up communication.</p>
<blockquote>
<p>Or some optimizers or networks do need to send some size (gradients * batch size)?</p>
</blockquote>
<p>No, batch size does not change the amount of traffic.</p>
<blockquote>
<p>For the RPC, the  <strong>Distributed Autograd</strong>  will take care of the communication in the backward direction at the RPC boundaries of each machine. So the communication happens when the user code calls .to_here for the  <strong>RREF</strong>  and the  <strong>Distributed Autograd</strong>  passes the gradients in backward?</p>
</blockquote>
<p>Yes, when the application calls <code>RRef.to_here()</code> in the <strong>forward</strong> pass, it inserts a <code>send</code> autograd function on the callee and a <code>recv</code> autograd function on the caller. These two functions contains some globally unique IDs to identify each other, and they help to connect multiple local autograd graphs into one global autograd graph. So that in the backward pass, when the autograd engine reaches the <code>recv</code> function, it knows where to propagate the gradient to, i.e., the <code>send</code> autograd function, and the backward can then proceed there. So from the application’s perspective, calling backward on the final loss will trigger a global backward across RPC boundaries. This <a href="https://pytorch.org/docs/master/rpc/distributed_autograd.html" rel="noopener nofollow ugc">distributed autograd note</a> might help to explain more.</p>
<blockquote>
<p>The traffic does relate to the batch size * the size of feature maps at the boundaries?</p>
</blockquote>
<p>No, batch size is still irrelevant here. This is because, the local autograd engine will only run a function once in the backward pass when all its input gradients are ready. More specifically, there is an input buffer for every autograd function. If a function <code>x</code> have multiple fanout in the forward pass, those fanout will accumulate gradient into the same input buffer and the local autograd engine will only trigger <code>x</code> when all gradients from those fanout are accumulated in the input buffer. So, the amount of data communicated through an RPC in the backward pass should be the same as the total tensor size went through that RPC in the forward pass.</p>

username: ZiyiZhu
cooked: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a>,</p>
<p>Thank you very much for the detailed explanations and they are very helpful.</p>
<p>Perhaps I do not understand it correctly or maybe I did not make it clear but I hope if you could explain more on your last comment in your previous reply.</p>
<aside class="quote no-group quote-modified" data-username="mrshenli" data-post="2" data-topic="79072">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<blockquote>
<p>The traffic does relate to the batch size * the size of feature maps at the boundaries?</p>
</blockquote>
<p>No, batch size is still irrelevant here. This is because, the local autograd engine will only run a function once in the backward pass when all its input gradients are ready. More specifically, there is an input buffer for every autograd function. If a function <code>x</code> have multiple fanout in the forward pass, those fanout will accumulate gradient into the same input buffer and the local autograd engine will only trigger <code>x</code> when all gradients from those fanout are accumulated in the input buffer. So, the amount of data communicated through an RPC in the backward pass should be the same as the total tensor size went through that RPC in the forward pass.</p>
</blockquote>
</aside>
<p>Take a nn.con2d as an example(<a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Conv2d" rel="noopener nofollow ugc">nn.Conv2d</a>). If this is the one at the RPC boundary and in the forward pass, the total tensor size to be transmitted to the following node should be equal to <strong>Batchsize</strong> x <strong>Channels</strong> x<strong>W</strong> x <strong>H</strong>, IIUC. So in the backward path, should it be the same size as this? Or it is accumulated as  <strong>Channels</strong> x<strong>W</strong> x <strong>H</strong>?</p>
<p>Thanks.</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="ZiyiZhu" data-post="3" data-topic="79072">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/z/ce7236/48.png" class="avatar"> ZiyiZhu:</div>
<blockquote>
<p>Take a nn.con2d as an example(<a href="https://pytorch.org/docs/stable/nn.html#torch.nn.Conv2d" rel="noopener nofollow ugc">nn.Conv2d</a>). If this is the one at the RPC boundary and in the forward pass, the total tensor size to be transmitted to the following node should be equal to <strong>Batchsize</strong> x <strong>Channels</strong> x <strong>W</strong> x <strong>H</strong> , IIUC. So in the backward path, should it be the same size as this? Or it is accumulated as <strong>Channels</strong> x <strong>W</strong> x <strong>H</strong> ?</p>
</blockquote>
</aside>
<p>My bad. If there is a batch dimension in the RPC argument tensor, then yes, the batch size will have an impact on the data size sent over wire. It will be B * C * W * H.</p>

username: ZiyiZhu
cooked: <p>Sure. Thank you for the detailed explanations.</p>
<p>Best,<br>
Ziyi</p>