Distributed Data Parallel Param Sync in Forward

username: Vibhatha_Abeykoon
cooked: <p>Hi,</p>
<p>I have a question about the DistributedDataParallel Module.</p>
<p>In the forward function call of the DDP, there is a <code>sycn_param</code> call which broadcasts the model parameters from rank 0 to all the other ranks to <code>keep the same state</code> of the model across all processes.</p>
<p><a href="https://github.com/pytorch/pytorch/blob/7f73f1d591afba823daa4a99a939217fb54d7688/torch/nn/parallel/distributed.py#L440" rel="nofollow noopener">Forwad Function in DDP</a></p>
<p><a href="https://github.com/pytorch/pytorch/blob/7f73f1d591afba823daa4a99a939217fb54d7688/torch/nn/parallel/distributed.py#L487" rel="nofollow noopener">Sync_call</a></p>
<p>I want to clarify the actual use of this function call.</p>
<p>Think of a case where all processes started and did initialize the model using a set of known weights. Then across all processes, the weights are uniform. In such a case is this useful? Because in every forward call, this synch is called and it could slow down the training. Please correct me if I am wrong.</p>
<p>Another case is where a user wants to add a mutation to the model weights in each process after a synch call (all-reduce). Think of an instance where a mutation helps to discover diversity in the training. And the all-reduce step ensembles this diversity. In such cases, having this call sync_param would cancel the effect the user expects?</p>
<p>What is the main expectation of the sync call? I referred to the <a href="https://pytorch.org/docs/stable/notes/ddp.html" rel="nofollow noopener">docs</a>, but I couldn’t get a clear picture.</p>
<p>Thank You,<br>
Vibhatha.</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/vibhatha_abeykoon">@Vibhatha_Abeykoon</a></p>
<p>That <code>_sync_params</code> call is there for two purposes:</p>
<ol>
<li>Intra rank/process parameter sync: this is only for the legacy single-process multi-device use case, where each process operates on multiple model replicas. And this is not a recommended way to use DDP.</li>
<li>Inter rank/process <strong>buffer</strong> sync: this does not sync parameters, and this will be skipped if your model does not have buffers (e.g., <code>running_mean</code> in <code>BatchNorm</code> layers).</li>
</ol>
<blockquote>
<p>What is the main expectation of the sync call?</p>
</blockquote>
<p>For many use cases, that <code>_sync_params</code> will be a no-op.</p>

username: Vibhatha_Abeykoon
cooked: <p><a class="mention" href="/u/mrshenli">@mrshenli</a>. Thank you for the response.</p>
<p>I see. The first point is clear to me.</p>
<p>About the second point, so this would be the case to support the functionality<br>
of some specific layers like <code>running_mean</code> in <code>BatchNorm</code> layers.  But for<br>
general cases, this is also skipped, so there won’t be such synchs.</p>
<p>Is this an assumption that we can make when we use DDP?</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="Vibhatha_Abeykoon" data-post="3" data-topic="88579">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vibhatha_abeykoon/48/17261_2.png" class="avatar"> Vibhatha_Abeykoon:</div>
<blockquote>
<p>About the second point, so this would be the case to support the functionality<br>
of some specific layers like <code>running_mean</code> in <code>BatchNorm</code> layers. But for<br>
general cases, this is also skipped, so there won’t be such synchs.</p>
<p>Is this an assumption that we can make when we use DDP?</p>
</blockquote>
</aside>
<p>Yep, this is correct.</p>