DistributedDataParallel training not efficient

AIME_team: <p>Hi,<br>
in our project using multiple gpus for training a resnet50 model with PyTorch and DistributedDataParallel, I encountered a problem. Here is the github-link for our project.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/aime-team/pytorch-benchmarks">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/aime-team/pytorch-benchmarks" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/90f03623971e003912114b8e472229fd51300e230168ec21f74a3d5229b4d6b9/aime-team/pytorch-benchmarks" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/aime-team/pytorch-benchmarks" target="_blank" rel="noopener nofollow ugc">GitHub - aime-team/pytorch-benchmarks: A benchmark framework for Pytorch</a></h3>

  <p>A benchmark framework for Pytorch. Contribute to aime-team/pytorch-benchmarks development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
Looking at the comparison of the validation accuracy progress after each epoch between a single GPU and multiple GPUs, it looks like the GPUs don’t share their training results with each other and it’s actually just one GPU training the model.<br>
Here is a comparison between a single Nvidia RTX 3090 and 4 Nvidia RTX 3090 using the same training parameters trained with the imagenet dataset.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/7/17694f0597d11deca29442b983f9ec8a749490b4.png" alt="4v_1GPU_training" data-base62-sha1="3l6Bub04lIcnil069g3HwGAjE4Q" width="450" height="300"></p>
<p>If the epoch axes of the 4x3090 graph is divided by the number of gpus (green curve), the graph has a similar shape like the one of the single GPU (blue curve). So basically training with 4 GPUS needs 4 epochs to get the same results like a single GPU achieves in only 1 epoch. Since the dataset is distributed over all GPUS, each GPU only uses 1/4 of the whole dataset, explaining the worse training result for each GPU after each epoch.<br>
Playing around with training parameters like using different batch sizes and learning rates gives me the same result. I have another comparison with different training parameters following in the next post because of the limit of embedded media for new users.</p>
<p>No matter what batch size or learning rate is used, training with a single GPU is always much more efficient then training with multiple GPUS using DistributedDataParallel.<br>
I also ran different tests with smaller datasets and different training parameters with the same effect.<br>
Using DataParallel instead of DistributedDataParallel works as expected: The training of multiple GPUS with a batchsize divided by the number of GPUS gives similar results as the training of a single GPU with the full batchsize.<br>
But since DataParallel is much slower than DistributedDataParallel, I would be very grateful, if someone could give me a hint what I am doing wrong.<br>
Thank you in advance</p>

AIME_team: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/4/84a610d6565a85d4740b4b83619221194f0b6af5.png" alt="imagenet_1vs2GPUs" data-base62-sha1="iVsORvygb5Ee9GuG0mu8AjoNcd7" width="480" height="360"></p>

wayi: <p>Very interesting project!</p>
<blockquote>
<p>So basically training with 4 GPUS needs 4 epochs to get the same results like a single GPU achieves in only 1 epoch.</p>
</blockquote>
<p>This is not true if you consider the sync among 4 GPUs per epoch. It should be equivalent to running 4 epochs on a single GPU.</p>
<p>Can you confirm if there is any communication between different processes (by printing the gradient values of different ranks after backward)? Gradients of different ranks should be the same after backward.</p>
<p>Additionally, your repo has an arg <code>average_gradients</code>. If you turn on this as a duplicate gradient averaging step, will it achieve the same accuracy as a single GPU?</p>

TinfoilHat0: <p>What about using <code>DataParallel?</code></p>

AIME_team: <p>Thank you for your replies!</p>
<aside class="quote no-group" data-username="wayi" data-post="3" data-topic="121624">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wayi/48/32374_2.png" class="avatar"> wayi:</div>
<blockquote>
<p>Can you confirm if there is any communication between different processes (by printing the gradient values of different ranks after backward)? Gradients of different ranks should be the same after backward.</p>
</blockquote>
</aside>
<p>I checked the gradient values of different ranks after the backward propagation and can confirm they are equal over all ranks. So the communication/sync between the GPUs seems to work. But I still don’t understand why the development of the validation accuracy is still that bad in our case. I wasn’t able to get a training setting where multiple GPUs using <code>DistributedDataParallel</code> gives a benefit over a single GPU. The epochs in a training with 4 GPUs are 4x shorter (because each GPU works with only 1/4 of the dataset) but we need 4x the amount of epochs to get the same result.</p>
<aside class="quote no-group" data-username="wayi" data-post="3" data-topic="121624">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wayi/48/32374_2.png" class="avatar"> wayi:</div>
<blockquote>
<p>Additionally, your repo has an arg <code>average_gradients</code>. If you turn on this as a duplicate gradient averaging step, will it achieve the same accuracy as a single GPU?</p>
</blockquote>
</aside>
<p>The option <code>average_gradients</code> I added as a try to solve this issue. But since the gradients of all ranks are already synced, it has no effect and the gradients remain unchanged.</p>
<aside class="quote no-group" data-username="TinfoilHat0" data-post="4" data-topic="121624" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tinfoilhat0/48/16880_2.png" class="avatar"> TinfoilHat0:</div>
<blockquote>
<p>What about using <code>DataParallel?</code></p>
</blockquote>
</aside>
<p>With <code>DataParallel</code> this issue doesn’t occur. That means training with multiple GPUs using a global batchsize divided by the number of GPUs (to have the same local batchsize) gives similar results as a single GPU with the full batchsize. But since <code>DataParallel</code> is significantly slower then <code>DistributedDataParallel</code> and we are trying to make multi-GPU-training as performant as possible, it would be awesome to solve this with <code>DistributedDataParallel</code>.</p>

wayi: <p>Have you tried DDP with both 1) and 2)?</p>
<ol>
<li>the same local batch size, which is equal to the global batch size divided by the number of GPUs,</li>
<li>the original learning rate multiplied by the number of GPUs.</li>
</ol>

quazi: <p>having the exact same issue, only monitoring training accuracy/loss. Also, training is slower that DataParallel and even single GPU. Wondering if/how you solved this. On a related note, would really appreciate a pointer to a detailed discussion/documentation of DistributedDataParallel… so far, I’m learning by making mistakes since the tutorials don’t seem to cover it comprehensively (or I haven’t found them yet).</p>

mrshenli: <blockquote>
<p>The epochs in a training with 4 GPUs are 4x shorter (because each GPU works with only 1/4 of the dataset) but we need 4x the amount of epochs to get the same result.</p>
</blockquote>
<blockquote>
<p>With <code>DataParallel</code> this issue doesn’t occur.</p>
</blockquote>
<p>Putting these two together, looks like the loss function might play a role here. With DDP, the gradient synchronization only occurs during the backward pass after loss computation, which means that each process/GPU independently computes the loss using its local input split. In contrast, DataParallel does not have this problem, as forward output is first gathered and then the loss is computed over all input data in that iteration. Will this make a difference in your application?</p>
<p>Another thing is that, when switching from single GPU to DDP-based multi-GPU training, you might need to tune the configs like learning rate to get the best result. See discussions in this post: <a href="https://discuss.pytorch.org/t/should-we-split-batch-size-according-to-ngpu-per-node-when-distributeddataparallel/72769" class="inline-onebox">Should we split batch_size according to ngpu_per_node when DistributedDataparallel</a></p>

AIME_team: <p>Past weeks I ran a few more tests,  but I still wasn’t able to solve this issue. Here are some training results with different batch sizes and learning rates:</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/c/9/c9af94dde546d28d4fffd2e1da89e4268bcf9614.png" alt="2vs1GPU" data-base62-sha1="sMc9uWN3bvo7DC8IXFmMkAKeHLm" width="640" height="480"></p>
<p>I also tried the ratios of the learning rate and batch size between single GPU and multi GPU training suggested by <a class="mention" href="/u/wayi">@wayi</a> . The single GPU curve is still always ahead of the multi GPU curve.<br>
Am I still missing something or is <code>DistributedDataParallel</code> just not working properly? Can please someone confirm its actually possible to achieve a similar progression of the validation accuracy in multi-GPU training using <code>DistributedDataParallel</code> compared to a single GPU training?</p>

Pavel: <p>Hi! I’ve faced the same issue. The solution was to increase learning rate proportional to the increase in the total batch size - as explained in details <a href="https://towardsdatascience.com/why-parallelized-training-might-not-be-working-for-you-4c01f606ef2c" rel="noopener nofollow ugc">in this great article</a>.<br>
In particular when I’ve used same learning rate for training on 1 GPU and 4 GPUs - there was no speed up at all. But when I’ve multiplied the learning rate by 4 for the 4 GPUs case - it converged much faster. The speed up was almost 4x - linear to the number of GPUs actually.</p>

hua_qiang: <aside class="quote no-group" data-username="AIME_team" data-post="1" data-topic="121624">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/bbe5ce/48.png" class="avatar"> AIME_team:</div>
<blockquote>
<p>No matter what batch size or learning rate is used, training with a single GPU is always much more efficient then training with multiple GPUS using DistributedDataParallel.</p>
</blockquote>
</aside>
<p>I am using DDP for model training, but the validate index not achieve  my expect while single GPU can achieve  my expect，even the time-consuming is shorter.</p>

JohnHerry: <p>A Key question. all plots above are about Validation-Acc, on dev set, what is the result on a big enough test set?</p>
<p>A single-GPU training may be more easisly to get a better Acc result, but multi-GPU will get a more smooth result. The all_reduce will get a “everage performance”.  not so sharp, but stable.</p>
<p>And if the training data size is very large, and training input condition is much more complex then a simple classifier, You may see stable converge on DDP, but not on single-card.</p>
<p>I think, whether you should chose single-card or DDP,  is decided on many situations, eg. the task your are training,   the size of trainging dataset,  the complexity of training inputs, and the traning strategy.</p>