Multi GPU Training is out of sync

username: omorrison1
cooked: <p>Hello,</p>
<p>I am in the process of training a PyTorch mode across multiple GPUs (Using DDP).</p>
<p>I had been under the impression that synchronisation happened automatically and Epochs appeared to occur at approximately the same time on each GPU.</p>
<p>However, upon running longer jobs, I have found that the two GPUs gradually become out of sync. With One GPU still processing much earlier epochs than the other.</p>
<p>I have tried adding torch.cuda.synchronize at the end of each epoch, but this doesn’t appear to make any difference at all.</p>
<p>Do you have any advice on how to move forward with this problem?</p>
<p>Thanks,<br>
Oliver</p>

username: ptrblck
cooked: <p><code>DDP</code> synchronizes the gradients in the backward pass (via buckets) and the <code>optimizer.step()</code> call is performed on all ranks before the next training iteration starts to make sure the models are at the same state. Do you see a different in model parameters during the forward pass?</p>

username: omorrison1
cooked: <p>The model is rather large.</p>
<p>I am not entirely sure how to best monitor the parameters.</p>
<p>Is there a simple way to check if the state has changed?</p>
<p>As a quick check I have printed the sum of the parameters before and after optimiser.step()  on each GPU:</p>
<p>[19:08:02] [GPU 1] (Epoch 0)  BEFORE 1256.03098<br>
[19:08:03] [GPU 1] (Epoch 0)  AFTER    1256.06498<br>
[19:08:03] [GPU 0] (Epoch 0)  BEFORE 1256.03098<br>
[19:08:03] [GPU 0] (Epoch 0)  AFTER    1256.06498</p>
<p>[19:08:05] [GPU 1] (Epoch 1)  BEFORE 1256.06498<br>
[19:08:05] [GPU 1] (Epoch 1)  AFTER    1256.12020<br>
[19:08:07] [GPU 0] (Epoch 1)  BEFORE 1256.06498<br>
[19:08:07] [GPU 0] (Epoch 1)  AFTER    1256.11947</p>
<p>[19:08:09] [GPU 1] (Epoch 2)  BEFORE 1256.12020<br>
[19:08:09] [GPU 1] (Epoch 2)  AFTER    1256.28199<br>
[19:08:11] [GPU 0] (Epoch 2)  BEFORE 1256.11947<br>
[19:08:11] [GPU 0] (Epoch 2)  AFTER    1256.27338</p>
<p>Thanks,<br>
Oliver</p>

username: omorrison1
cooked: <blockquote>
<p>and the <code>optimizer.step()</code> call is performed on all ranks before the next training iteration starts</p>
</blockquote>
<p>It doesn’t appear to be the case for me?</p>
<p>I have been following the approach found here: <a href="https://pytorch.org/tutorials/beginner/ddp_series_multigpu.html?utm_source=youtube&amp;utm_medium=organic_social&amp;utm_campaign=tutorial" class="inline-onebox" rel="noopener nofollow ugc">Multi GPU training with DDP — PyTorch Tutorials 2.3.0+cu121 documentation</a></p>
<p>Optimizer.step is called at different times on each GPU depending on when the GPU happens to get to that line in the code and the parameter updates don’t seem to be communicated to other GPUs.</p>

username: ptrblck
cooked: <p>Parameter updates won’t be communicated, but the gradients will be all-reduced before the <code>optimizer.step()</code> call is performed as explained before.</p>
<p>Your “sum-check” indicates the parameters are almost equal. So far nothing points to the GPUs being out of sync.</p>

username: omorrison1
cooked: <p>Sorry, I seem to have a fundamental misunderstanding about how this process works.</p>
<p>I was under the impression that the GPUs communicated so that each local copy of the model has the same gradients.</p>
<p>Then when the optimizer step is performed, the local copies of the model complete the epoch together reaching the same end state.</p>
<p>If each GPU produces produces an entirely distinct model with different parameter values, then what is the value of distributed training?</p>
<p>Currently, I am adding a while loop at the end of each epoch which checks to see if a checkpoint file exists for that epoch, because without that in place GPUs will drift many epochs apart.</p>
<p>Thanks,<br>
Oliver</p>

username: omorrison1
cooked: <p>A more pronounced example is seen here:</p>
<p>[14:22:36] [GPU 1] (Epoch 13)  ----End of Epoch 	| 1061.04428,</p>
<p>[14:22:39] [GPU 0] (Epoch 13)  ----End of Epoch 	| 1200.90550,</p>
<p>The sum of parameters is much further apart and GPU 1 starts the next epoch 3 seconds before GPU 0.</p>
<p>If I ended training at this point (the end of epoch 13), which model would I save? If I save the model on GPU 0, then would I not be losing all the training that took place on GPU 1?</p>
<p>Thanks,<br>
Oliver</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="omorrison1" data-post="6" data-topic="205185">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/omorrison1/48/70586_2.png" class="avatar"> omorrison1:</div>
<blockquote>
<p>I was under the impression that the GPUs communicated so that each local copy of the model has the same gradients.</p>
</blockquote>
</aside>
<p>This is correct and is also explained in the <a href="https://pytorch.org/docs/master/notes/ddp.html">design docs</a>.</p>
<aside class="quote no-group" data-username="omorrison1" data-post="6" data-topic="205185">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/omorrison1/48/70586_2.png" class="avatar"> omorrison1:</div>
<blockquote>
<p>Then when the optimizer step is performed, the local copies of the model complete the epoch together reaching the same end state.</p>
</blockquote>
</aside>
<p>This is also correct as explained before and shown by the output in your previous post.</p>
<aside class="quote no-group" data-username="omorrison1" data-post="6" data-topic="205185">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/omorrison1/48/70586_2.png" class="avatar"> omorrison1:</div>
<blockquote>
<p>If each GPU produces produces an entirely distinct model with different parameter values, then what is the value of distributed training?</p>
</blockquote>
</aside>
<p>That’s not the case. Since the parameters and gradients are (almost) equal, the (almost) same models will be created up to floating point precision errors.</p>
<p>If you have a reproduction, showing this is not the case, please post it.</p>

username: omorrison1
cooked: <p>Unfortunately, my code base has grown rather complicated, so I don’t have a convenient reproduction of this strange behaviour.</p>
<p>I use “model.module.forward” in place of “model.forward”, is it possible that this is the reason why the synchronisation is not taking place?</p>
<p>If jobs a run for long enough one GPU will get many epochs ahead of the other, so I know that they aren’t performing these optimisation steps together.</p>
<p>Is there perhaps some way to monitor these “buckets”, to check that the gradients are actually being broadcast and received?</p>
<p>Thanks,<br>
Oliver</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="omorrison1" data-post="9" data-topic="205185">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/omorrison1/48/70586_2.png" class="avatar"> omorrison1:</div>
<blockquote>
<p>I use “model.module.forward” in place of “model.forward”, is it possible that this is the reason why the synchronisation is not taking place?</p>
</blockquote>
</aside>
<p>Yes, if you directly access the internal <code>.module</code> directly, you will skip the DDP logic as it depends on the direct model call via <code>output = model(input)</code></p>

username: omorrison1
cooked: <p>I have been using “model.module.forward”, because when I use “model.forward” I encounter an error.</p>
<p>The error is raised when I reach the backwards pass and always states that the “version” of one of the tensors is one more than it should be. (see example below)</p>
<pre><code class="lang-auto">RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.DoubleTensor [87, 4]], which is output 0 of AddBackward0, is at version 1; expected version 0 instead.
</code></pre>
<p>This error is avoided is I am using a single GPU or if I am using “model.module.forward”, so I guess the unexpected modification is somehow linked to DDP?</p>
<p>EDIT:</p>
<p>In particular, the error is not raised when the first GPU reaches the backwards pass. It is raised when the second GPU reaches the backwards pass.</p>
<p>Thanks,<br>
Oliver</p>

username: ptrblck
cooked: <p>You would need to narrow down which tensor is modified inplace and how it correlates with the usage of <code>DDP</code>. From the error message I wouldn’t know which operation is causing it and why it’s specific to <code>DDP</code>.</p>

username: nickums
cooked: <p>suggest you use ‘Accelerate’, it automatically distributes torch processing over available GPUs and has features for synchronising the parallel processing. See <a href="https://huggingface.co/docs/transformers/perf_train_gpu_one#using--accelerate" class="inline-onebox" rel="noopener nofollow ugc">Methods and tools for efficient training on a single GPU</a><br>
<a href="https://huggingface.co/docs/accelerate/basic_tutorials/overview" class="inline-onebox" rel="noopener nofollow ugc">Overview</a><br>
Nick</p>