Stucks on 8gpu training setting

pizza: <p>Hello,</p>
<p>I am using Pytorch (Pytorch Lightning Framework) to train the Text to Text Transformer model (<a href="https://huggingface.co/google/mt5-base/tree/main" class="inline-onebox" rel="noopener nofollow ugc">google/mt5-base at main</a>).</p>
<p>I trained them on 1, 4, 5, 8 gpu environment using DDP.<br>
However, all of 8gpu and 5gpu training attempts, are stuck and failed at a specific point in a specific epoch (54).</p>
<p>This is the last log before stuck, as it seems, its end of an epoch, so I assume that training is stuck due to data loading for next epoch in 8gpu or 5gpu environment.</p>
<p>This issue also occurred regardless of <code>num_worker</code> in <code>DataLoader</code> or different batch_size (32, 16)</p>
<pre><code class="lang-auto">Epoch 54: 100%|█████████▉| 2921/2931 [43:38&lt;00:08,  1.12it/s, loss=.., v_num=0]
Epoch 54: 100%|█████████▉| 2925/2931 [43:41&lt;00:05,  1.12it/s, loss=.., v_num=0]
Validating:  99%|█████████▉| 280/282 [02:32&lt;00:01,  1.59it/s]e[A
Epoch 54: 100%|██████████| 2931/2931 [44:01&lt;00:00,  1.11it/s, loss=.., v_num=0]
</code></pre>
<p>Any comment or suggestion would be appreciated.</p>
<p>Thank you.</p>

pritamdamania87: <p>Do you have even number of batches across all GPUs? If not, the training could get stuck since DDP performs a collective sync across all GPUs in the backward pass. If you have uneven batches, you can try using the <a href="https://pytorch.org/docs/stable/generated/torch.nn.parallel.DistributedDataParallel.html#torch.nn.parallel.DistributedDataParallel.join" rel="noopener nofollow ugc">join</a> API to get around this.</p>

pizza: <p><a class="mention" href="/u/pritamdamania87">@pritamdamania87</a>,</p>
<p>First of all, thank you for your reply.</p>
<ul>
<li>If I understand your comment correctly, It seems I both had training with even and uneven batch size across all GPUs.</li>
<li>The log shows 2931 which is the uneven number of batches assigned for a single GPU. however, in another training session, a even number of batches (like 1466) have also experienced the same issue.</li>
<li>Could you suggest how to correctly measure the number of batches per GPU? in case I was wrong in measure?</li>
<li>I will check to join API try to get around this issue of course.</li>
</ul>
<p>Thank you.!</p>

pritamdamania87: <blockquote>
<ul>
<li>Could you suggest how to correctly measure the number of batches per GPU? in case I was wrong in measure?</li>
</ul>
</blockquote>
<p>You would probably have to add some logging code to your application which records the number of batches on each rank as part of the training loop.</p>

pizza: <p><a class="mention" href="/u/pritamdamania87">@pritamdamania87</a></p>
<p>Thank you for your comment <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I am now still in debug, and experienced the same error on DataParallel setting (without hang; just training loop stops at the end of 54 epoch) <img src="https://discuss.pytorch.org/images/emoji/apple/cry.png?v=9" title=":cry:" class="emoji" alt=":cry:"></p>
<p>Is this good evidence for the issue?</p>

pizza: <p>This issue is caused by my code mistake.<br>
I accidentally give a small (54) to max epoch value, after adjust the max epoch value, works fine on DP or CPU environment.<br>
DDP also hangs after the max epoch which seems is interpretable behavior, but not normal behavior. (currently I don’t know why.)</p>