How Can DDP Processes Get Out of Sync?

paepcke: <p>I run a single machine, 2-GPU resnet-based training with one process for each GPU. Each process prints a msg at the start of each epoch.</p>
<p>After a few minutes, one process draws ahead of the other. The difference ends up to be multiple epochs, even though both processes do finish.</p>
<p>How can this speed difference occur, given that DDP synchronizes the two processes during each call to <code>backward()</code>?</p>
<p>Based on a note on a Web site about different input sizes to different DDP processes I tried</p>
<pre><code class="lang-auto">with model.join():
    &lt;training loop&gt;
</code></pre>
<p>But observed no different behavior. How does DDP sneak past that back prop synchronization point?</p>
<p>I unfortunately could not reproduce the problem on a simple case. But maybe I am missing some logic?</p>
<pre><code class="lang-auto">- Ubuntu 20.04
- Pytorch torch-1.7.1-py3.8
- torch.cuda.nccl.version(): 2708
- 2xNvidia GTX Titan
- Single machine, 2 process, one for each of the GPUs
</code></pre>

rvarm1: <p>Hi, unless an explicit synchronization (such as with a device to host copy or torch.cuda.synchronize) is triggered, this sort of skew is possible due to the different GPUs, but such a significant skew (many epochs) seems unlikely. Are there any other jobs/processes running on your GPU that may slow this down and is the problem consistently reproducible?</p>
<p>As far as <code>model.join()</code> that API is to support different processes having different no. of inputs, is that your case here? If not, you would not need <code>model.join()</code>.</p>

paepcke: <p>Thank you for both answers, Rohan! Before responding<br>
I was trying to verify that moving the model save to after</p>
<p>the optimizer step solved the issue. And I didn’t have a<br>
chance for that yet.</p>
<p>The oddity with the out-of-synch processes is that they run in lockstep<br>
for quite a while, meaning several minutes. And then one of</p>
<p>them slows way down. Often, once the faster process finishes,</p>
<p>the slow one sits with 100% of both GPU and CPU (according to top).</p>
<p>So something is seriously wrong.</p>
<p>What confuses me is that I thought DDP will force synchronization</p>
<p>as part of the backward() call to ensure the same gradients</p>
<p>being used on all the processes going forward. So how could two</p>
<p>processes get out of sync at all even just beyond a single train loop.</p>
<p>Nobody is running on the machine other than these two processes.</p>
<p>Inputs should be of the same lengths; the model.join() attempt was<br>
just out of desperation. I run with drop-last in the dataloader, so even<br>
the final batch is fully populated. If anything, the two processes are <em>too</em></p>
<p>decoupled, rather than one waiting for the other due to unequal inputs.</p>
<p>The one difference from all the tutorial setups is that I am using</p>
<p>k-fold cross validation with a distributed sampler. So epochs are<br>
put together by train/validate over multiple splits (i.e. rotating the</p>
<p>validation folds) But to the DDP mechanism that shouldn’t make a difference,</p>
<p>I would think.</p>
<p>For now both processes run on the same machine to exclude NCCL version mismatch.</p>
<p>Though my goal is cross-machine work. I did try updating NCCL and cuda to (I think)</p>
<p>version 11? But this measure did not fix the issue.</p>
<p>Does any of the above point to a misunderstanding on my part? Of course,<br>
it could simply be a bug in the code somewhere.</p>
<p>Andreas</p>

rvarm1: <aside class="quote no-group" data-username="paepcke" data-post="1" data-topic="109478">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/paepcke/48/28444_2.png" class="avatar"> paepcke:</div>
<blockquote>
<p>The difference ends up to be multiple epochs, even though both processes do finish.</p>
</blockquote>
</aside>
<p>The way that NCCL allreduce calls (basically the backward sync point you mention)  work is by enqueuing the ops on GPU, and then the CPU host can continue, so while unlikely this desynchronization can occur.</p>
<p>Are you noticing any significant performance issues when this happens? If you do need to synchronize the GPUs, you can use <code>torch.cuda.synchronize()</code> or <code>dist.barrier()</code>, though this might affect the performance, especially if called very frequently.</p>

paepcke: <p>Sorry, Rohan; I needed to move forward for now, and</p>
<p>will run on a single GPU for now. I did try the synchronize()<br>
and barrier(), but somehow one process ends up taking<br>
100% of a CPU, and all memory on its GPU. So something<br>
is wrong; I’ll have to go through my own code again when I<br>
get the chance. Thank you nonetheless!</p>