Getting uneven GPU utilization using nn.dataparallel and how to place loss in data parallel

bing: <p>I am trying to use 3 GPU in a single machine using `nn.DataParallel using following way -</p>
<pre><code class="lang-auto">device =  torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
if torch.cuda.device_count() &gt; 1:
        print("Let's use", torch.cuda.device_count(), "GPUs!")
       model_transfer = nn.DataParallel(model_transfer,device_ids=range(torch.cuda.device_count()))
model_transfer.to(device)
</code></pre>
<p>I am getting uneven GPU utilization, i.e GPU1 : 100% and rest of them 0%.<br>
Where I am doing wrong?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/0/00be182eeb76c9822b94a04de6fd9784be86029a.png" data-download-href="https://discuss.pytorch.org/uploads/default/00be182eeb76c9822b94a04de6fd9784be86029a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/00be182eeb76c9822b94a04de6fd9784be86029a_2_690x335.png" alt="image" data-base62-sha1="6zh54Vk8zi4xLHlLnao83T2P7s" width="690" height="335" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/00be182eeb76c9822b94a04de6fd9784be86029a_2_690x335.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/0/00be182eeb76c9822b94a04de6fd9784be86029a_2_1035x502.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/0/00be182eeb76c9822b94a04de6fd9784be86029a.png 2x" data-dominant-color="FCFCFD"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1352×658 20.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>GPU memory utilization is same on all of them-<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/7/e7fda13efbc71b669faade90c640a2600e7baaf1.png" data-download-href="https://discuss.pytorch.org/uploads/default/e7fda13efbc71b669faade90c640a2600e7baaf1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7fda13efbc71b669faade90c640a2600e7baaf1_2_689x336.png" alt="image" data-base62-sha1="x6hH6bGwRlHsMcSn3ZYKVLBxhND" width="689" height="336" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7fda13efbc71b669faade90c640a2600e7baaf1_2_689x336.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/7/e7fda13efbc71b669faade90c640a2600e7baaf1_2_1033x504.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/e/7/e7fda13efbc71b669faade90c640a2600e7baaf1.png 2x" data-dominant-color="FDFCFC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1344×656 15.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>On searching further, I come across this answer <a href="https://discuss.pytorch.org/t/uneven-gpu-utilization-during-training-backpropagation/36117/14" class="inline-onebox">Uneven GPU utilization during training backpropagation - #14 by colllin</a></p>
<p>Now, the thing is they are using <code>CriterionParallel(loss)</code> for custom class, how to do it for normal <code>dataloader class</code></p>