Model and optimizer parameters synchronization at start

whatdhack: <p>I am reading the page <a href="https://pytorch.org/docs/stable/notes/ddp.html" class="inline-onebox" rel="noopener nofollow ugc">Distributed Data Parallel — PyTorch 2.5 documentation</a> . Although, it was written a while ago, I think the basic  still applies.   Looking at the example code in that page, I am wondering how the model and optimizer parameters  of each rank are made exactly same ? Else it would likely not converge, right ?</p>

ptrblck: <p>From the <a href="https://pytorch.org/docs/main/notes/ddp.html#internal-design">DDP Internal Design Doc</a>:</p>
<blockquote>
<p><strong>Construction</strong>: The DDP constructor takes a reference to the local module, and broadcasts <code>state_dict()</code> from the process with rank 0 to all other processes in the group to make sure that all model replicas start from the exact same state.</p>
</blockquote>

whatdhack: <p>Thanks. Follow on question is why this broadcast is needed ?  If the random seed/s are set to the same value, then should not the state of each rank be identical at the start ?</p>

ptrblck: <aside class="quote no-group" data-username="whatdhack" data-post="3" data-topic="214872">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/whatdhack/48/67702_2.png" class="avatar"> whatdhack:</div>
<blockquote>
<p>If the random seed/s are set to the same value, then should not the state of each rank be identical at the start ?</p>
</blockquote>
</aside>
<p>The user would need to make sure the same seeds are used and exactly the same order of calls into the pseudorandom number generator is made, which can be challenging. Broadcasting the <code>state_dict</code> from rank0 is a simple solution.</p>

whatdhack: <p>What could  cause differences in call to PNRG in the ranks ?   Would be good to have an idea of the sources of non-determinism.  Any effort in PyTorch to remove  non-determinism ?</p>

ptrblck: <aside class="quote no-group" data-username="whatdhack" data-post="5" data-topic="214872">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/whatdhack/48/67702_2.png" class="avatar"> whatdhack:</div>
<blockquote>
<p>What could cause differences in call to PNRG in the ranks ?</p>
</blockquote>
</aside>
<p>Conditional user code, e.g. shuffling dataset indices depending on the <code>rankId</code>, but of course the possibilities to make calls into the PRNG are unlimited.<br>
Expecting users to guarantee exactly the same calls into the PRNG are made is unreasonable and the solution is for a single broadcast of the <code>state_dict</code> at the beginning of the training causing negligible overhead.</p>
<aside class="quote no-group" data-username="whatdhack" data-post="5" data-topic="214872">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/whatdhack/48/67702_2.png" class="avatar"> whatdhack:</div>
<blockquote>
<p>Would be good to have an idea of the sources of non-determinism. Any effort in PyTorch to remove non-determinism ?</p>
</blockquote>
</aside>
<p>Different calls into the PRNG are not causing non-determinism, since the random numbers are still <em>deterministic</em> but different.<br>
If you want to produce deterministic results, please refer to the <a href="https://pytorch.org/docs/stable/notes/randomness.html">Reproducibility docs</a>.</p>

whatdhack: <p>Thanks. The   <a href="https://pytorch.org/docs/stable/notes/randomness.html" rel="noopener nofollow ugc">randomness doc </a> appears to  mainly address simple  single process apps.  It will be good to have something similar for a real app in a distributed environment.</p>